process failed: java.lang.ArrayIndexOutOfBoundsException

nolle

Hi folks,

I'm using RapidMiner 5.3.008 and get the following error on ApplyModel of a trained NeuralNetwork:

The setup does not seem to contain any obvious errors, but you should check the log messages or activate the debug mode in the settings dialog in order to get more information about this problem.

The error log contains:
SEVERE: Process failed: operator cannot be executed (DataRow: table index 82 of Attribute Draught_Lobby_Yes is out of bounds.). Check the log messages...
java.lang.ArrayIndexOutOfBoundsException: DataRow: table index 82 of Attribute Draught_Lobby_Yes is out of bounds.
at com.rapidminer.example.table.DataRow.get(DataRow.java:78)
at com.rapidminer.example.Example.getValue(Example.java:94)
at com.rapidminer.operator.learner.functions.neuralnet.InputNode.calculateValue(InputNode.java:61)
at com.rapidminer.operator.learner.functions.neuralnet.SigmoidFunction.calculateValue(SigmoidFunction.java:50)
at com.rapidminer.operator.learner.functions.neuralnet.InnerNode.calculateValue(InnerNode.java:72)
at com.rapidminer.operator.learner.functions.neuralnet.LinearFunction.calculateValue(LinearFunction.java:49)
at com.rapidminer.operator.learner.functions.neuralnet.InnerNode.calculateValue(InnerNode.java:72)
at com.rapidminer.operator.learner.functions.neuralnet.OutputNode.calculateValue(OutputNode.java:68)
at com.rapidminer.operator.learner.functions.neuralnet.ImprovedNeuralNetModel.performPrediction(ImprovedNeuralNetModel.java:192)
at com.rapidminer.operator.learner.PredictionModel.apply(PredictionModel.java:76)
at com.rapidminer.operator.ModelApplier.doWork(ModelApplier.java:100)
at com.rapidminer.operator.Operator.execute(Operator.java:855)
at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
在com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:379)
at com.rapidminer.operator.Operator.execute(Operator.java:855)
at com.rapidminer.Process.run(Process.java:949)
at com.rapidminer.Process.run(Process.java:873)
at com.rapidminer.Process.run(Process.java:832)
at com.rapidminer.Process.run(Process.java:827)
at com.rapidminer.Process.run(Process.java:817)
com.rapidminer.gui.ProcessThread.run (ProcessThread.java:63)

Jun 25, 2013 10:04:23 AM com.rapidminer.gui.ProcessThread run
SEVERE: Here: Process[1] (Process)
subprocess 'Main Process'
+- Read Database[1] (Read Database)
+- Set Role (2)[1] (Set Role)
+- Retrieve NewcastleAndMiddlesbrough[1] (Retrieve)
+- Select Attributes (2)[1] (Select Attributes)
+- Rename[1] (Rename)
+- Select Attributes (4)[1] (Select Attributes)
+- Generate Attributes[1] (Generate Attributes)
+- Set Role[1] (Set Role)
+- Real to Integer[1] (Real to Integer)
+- Text to Nominal[1] (Text to Nominal)
+- Multiply (2)[1] (Multiply)
+- Generate Attributes (2)[1] (Generate Attributes)
+- Nominal to Text[1] (Nominal to Text)
+- Set Role (3)[1] (Set Role)
+- Multiply[1] (Multiply)
+- Nominal to Numerical (2)[1] (Nominal to Numerical)
+- Weight by Correlation (2)[1] (Weight by Correlation)
+- Select Attributes (3)[1] (Select Attributes)
+- Select by Weights (2)[1] (Select by Weights)
+- Nominal to Numerical[1] (Nominal to Numerical)
+- NeuralNet(Correlation)[1] (Neural Net)
==> +- ApplyModel(Correliert)[1] (Apply Model)
+- Rename (2)[0] (Rename)
+- Join (2)[0] (Join)
+- Join[0] (Join)
+- Select Attributes[0] (Select Attributes)
Jun 25, 2013 10:04:23 AM com.rapidminer.gui.ProcessThread run
SEVERE: java.lang.ArrayIndexOutOfBoundsException: DataRow: table index 82 of Attribute Draught_Lobby_Yes is out of bounds.
java.lang.ArrayIndexOutOfBoundsException: DataRow: table index 82 of Attribute Draught_Lobby_Yes is out of bounds.
at com.rapidminer.example.table.DataRow.get(DataRow.java:78)
at com.rapidminer.example.Example.getValue(Example.java:94)
at com.rapidminer.operator.learner.functions.neuralnet.InputNode.calculateValue(InputNode.java:61)
at com.rapidminer.operator.learner.functions.neuralnet.SigmoidFunction.calculateValue(SigmoidFunction.java:50)
at com.rapidminer.operator.learner.functions.neuralnet.InnerNode.calculateValue(InnerNode.java:72)
at com.rapidminer.operator.learner.functions.neuralnet.LinearFunction.calculateValue(LinearFunction.java:49)
at com.rapidminer.operator.learner.functions.neuralnet.InnerNode.calculateValue(InnerNode.java:72)
at com.rapidminer.operator.learner.functions.neuralnet.OutputNode.calculateValue(OutputNode.java:68)
at com.rapidminer.operator.learner.functions.neuralnet.ImprovedNeuralNetModel.performPrediction(ImprovedNeuralNetModel.java:192)
at com.rapidminer.operator.learner.PredictionModel.apply(PredictionModel.java:76)
at com.rapidminer.operator.ModelApplier.doWork(ModelApplier.java:100)
at com.rapidminer.operator.Operator.execute(Operator.java:855)
at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
在com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:379)
at com.rapidminer.operator.Operator.execute(Operator.java:855)
at com.rapidminer.Process.run(Process.java:949)
at com.rapidminer.Process.run(Process.java:873)
at com.rapidminer.Process.run(Process.java:832)
at com.rapidminer.Process.run(Process.java:827)
at com.rapidminer.Process.run(Process.java:817)
com.rapidminer.gui.ProcessThread.run (ProcessThread.java:63)

Is this a known issue or did I missed something?

Cheers,
Andy

nolle

Please find below the XML of my process as well as the data:

<输出/ >


DB_TABLE_NAME
SAPPrediction


FIELD_NAME_0
ID


ROLE_0
id


FIELD_NAME_1



ROLE_1



FIELD_NAME_2



FIELD_NAME_3



FIELD_NAME_4



FIELD_NAME_5



FIELD_NAME_6



FIELD_NAME_7



FIELD_NAME_8



FIELD_NAME_9



ROLE_2



ROLE_3



ROLE_4



ROLE_5



ROLE_6



ROLE_7



ROLE_8



ROLE_9































































http://arcdev.housing.salle.url.edu/semanco/repository/sap/age_class/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/bottom_floor_type/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/type_of_wall/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/wall_insulation_type/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/roof_area/", ""))"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/roof_tilt/", ""))"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/draught_lobby/", "")"/>


















<参数键= value =“Bottom_Floor_Type replaceAll(Bottom_Floor_Type, " ", "_")"/>




























<参数键= "属性" value = " addressValue | Ag)e_Class|Bottom_Floor_Type|Draught_Lobby|Roof_Area|Roof_Tilt|Type_Of_Wall|Wall_Insulation_Type|building_PerimeterValue|ground_Floor_AreaValue|ground_Floor_HeightValue|number_Of_Complete_StoreysValue|number_Of_Natural_Ventilation_DeviceValue|number_Of_RoomsValue|number_Of_Sides_ShelteredValue|percentage_Of_Window_Door_Draught_StrippedValue|window_TypeValue"/>
























































<连接from_op = from_port“选择的权重(2)”="example set output" to_op="NeuralNet(Correlation)" to_port="training set"/>













[ /code]

Please let me know if you also need the data used in this process.

MariusHelf

Hi,

this is a very complex process, next time please break it down to the minimal operator set that still reproduces the error.

The problem here is probably that you train the model on data with different attributes than where you apply it. Please set a breakpoint before the Neural Net operator and the Apply Model operator to check that the attribute names and types match. If they don't you have to adapt the preprocessing such that it is the same for both training and application.

The next RapidMiner version will probably provide a more informative error message when you run into this problem.

Best regards,
Marius

nolle

Many thanks to your reply and sorry for that complex process (but all operators before the neural net can probably ignored because they are only for preprocessing and align both example sets).

I've compared the two example sets and the data as well as the meta data are equal. I've also stored both example sets to create a trimmed process that only train and apply the model. The strange thing is that in the original process the error still occurs but the trimmed process runs through, although in both processes the operators are equally configured.
Another point is that the results for the trimmed processes depends on the used example set even though the example sets looks equally.
Please find both processes and example sets on this link:https://www.dropbox.com/s/30nbcynzi6gtgw4/testdata-dump.zip

Cheers,
Andy

MariusHelf

Andy,

这是什么类型的数据?它没有任何file endings or whatsoever.

The easiest way to provide me with data and processes would be to:
- store the data and the processes in a single repository folder
- configure all processes such that I can run them right away (that means they only use data from that folder)
- rightclick the repository folder and select Show in File Browser
- upload the respective file system folder as zip.

Best regards,
Marius

nolle

You can find the requested data at:https://www.dropbox.com/s/wxtnpz98q562g2w/testdata.zip

MariusHelf

Thank you for this detailed processes and data. Now I could reproduce the error, and it is clearly a bug. It probably relates to the bug described below (that has already been fixed), but I'll forward it to our developers in any case.



Still, what I have said is valid: the data sets have different attributes and meta data. Even though it seems at a quick glance that all attributes of the training data are present in the test data, there are many additional attributes in the test data, and the order of the attributes is different. For most algorithms this is not a problem, but the Neural Net has a bug in the current version such that it requires the exact same attributes in the same order as in the training data.
This bug has already been fixed and will be part of the next release.
In the meanwhile, please use Select Attributes to remove the additional features, and Reorder Attributes to equalize the order. The easiest way here would be to apply Reorder Attributes with the default settings on both data sets.
Additionally, you can workaround the original error message by storing and retrieving the training data. I have modified your process and attached it below.

Sorry for the inconvenience - the next release will bring relieve

Best regards,
Marius

MariusHelf

Process Pt.1

<输出/ >


DB_TABLE_NAME
SAPPrediction


FIELD_NAME_0
ID


ROLE_0
id


FIELD_NAME_1



ROLE_1



FIELD_NAME_2



FIELD_NAME_3



FIELD_NAME_4



FIELD_NAME_5



FIELD_NAME_6



FIELD_NAME_7



FIELD_NAME_8



FIELD_NAME_9



ROLE_2



ROLE_3



ROLE_4



ROLE_5



ROLE_6



ROLE_7



ROLE_8



ROLE_9




























































http://arcdev.housing.salle.url.edu/semanco/repository/sap/age_class/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/bottom_floor_type/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/type_of_wall/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/wall_insulation_type/", "")"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/roof_area/", ""))"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/roof_tilt/", ""))"/>
http://arcdev.housing.salle.url.edu/semanco/repository/sap/draught_lobby/", "")"/>





















<参数键= value =“Bottom_Floor_Type replaceAll(Bottom_Floor_Type, " ", "_")"/>
































<参数键= "属性" value = " addressValue | Ag)e_Class|Bottom_Floor_Type|Draught_Lobby|Roof_Area|Roof_Tilt|Type_Of_Wall|Wall_Insulation_Type|building_PerimeterValue|ground_Floor_AreaValue|ground_Floor_HeightValue|number_Of_Complete_StoreysValue|number_Of_Natural_Ventilation_DeviceValue|number_Of_RoomsValue|number_Of_Sides_ShelteredValue|percentage_Of_Window_Door_Draught_StrippedValue|window_TypeValue"/>

MariusHelf

Process Pt. 2

<连接from_op = from_port“选择的权重(2)”="example set output" to_op="Data to Weights" to_port="example set"/>

nolle

Many thanks for that!

nolle

By the way: is it possible to get the current trunk version of RapidMiner that already includes the fixed bug? I've already checked out the current RapidMiner_Unuk version but the last checkin was in February....
Our problem is that we use RapidMiner and RapidAnalytics in our research project and it would be helpful to have the bug fixed instead of integrate some work arounds. Or can you tell us the next release day?

Skirzynski

We have no concrete date for the next release and as you already mentioned the source code on Sourceforge is not up-to-date since we have switched to git internally. However, we plan to push to our GitHub account in the near future. I will come back to this thread when this is done.

Nils_Woehler

A new release is out since some days now and the code for this release can be found here:https://github.com/rapid-i/rapidminer