SOM Reduction - Preprocessing Save doesn't work

Legacy User

When I attempt to create and save a preprocessing model with 205 features ( a mix of nominal and numeric features ) with about 10,000 training rows, after processing for about a hour, the modelwrite writes out about a 12 meg model, then hangs and freezes. Any attempts to use this model results in errors. Is this a size or complexity limitation?

TobiasMalbrecht

Hi,

which preprocessing model did you create and save? Could you please post your process XML?! Given your description, I have no idea what could be the problem here.

Regards,
Tobias

land

Hi,
since the SOMReduction should save the generated SOM, there should be no complexity problem with the number of examples. (Ignoring time needed to compute the SOM)
How long did it freeze, before it terminated the process? It might be, that saving might last a few minutes.

A quoting of the error messages would be usefull, too.


Greetings,
Sebastian

Legacy User

Here's the error message when trying to load the model created from the SOM preprocessing

Exception: com.rapidminer.operator.UserError
Message: Could not read file 'F:\ModelDevelopment\preprocess_som.mod': Cannot read from XML stream, wrong format: Could not call java.security.CodeSource.readObject()
---- Debugging information ----
message : Could not call java.security.CodeSource.readObject()
cause-exception: . lang。RuntimeException
cause-message : null
class : com.rapidminer.operator.ContainerModel
required-type : java.security.CodeSource
line number : 2224478
-------------------------------.
Stack trace:

com.rapidminer.operator.io.ModelLoader.apply(ModelLoader.java:97)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.Process.run(Process.java:571)
com.rapidminer.Process.run(Process.java:541)
com.rapidminer.Process.run(Process.java:531)
com.rapidminer.gui.ProcessThread.run(ProcessThread.java:61)

Cause
Exception: java.io.StreamCorruptedException
Message: invalid stream header: 1F8B0800
Stack trace:

java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
java.io.ObjectInputStream.(ObjectInputStream.java:280)
com.rapidminer.operator.io.ModelLoader.apply(ModelLoader.java:76)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
com.rapidminer.operator.Operator.apply(Operator.java:656)
com.rapidminer.Process.run(Process.java:571)
com.rapidminer.Process.run(Process.java:541)
com.rapidminer.Process.run(Process.java:531)
com.rapidminer.gui.ProcessThread.run(ProcessThread.java:61)

land

Hi,
I just found your bug. Funny oneSince the SOM was a thread itself, the IOObjectWriter tried to save nearly everything. Complete rapidMiner, libraries, everything, since the thread knew its threadpool, which probably knew all threads including the gui...And when the IOObjectWriter started to save itself, it hang up. What was probably a good idea after all

In the newest repository version it now works.

Greetings,
Sebastian

Legacy User

Great. How or where do I access the repository version? I assume it's CVS somewhere, and is the repository built into binaries or do I have to compile ... ?

TobiasMalbrecht

Hi,

http://rapid-i.com/content/view/25/48/lang,de/

explains the anonymous CVS access to our developer version via Eclipse. The developer CVS repository naturally only contains the source code. The compilation or running of RapidMiner, respectively, is pretty easy using Eclipse. Just follow the instructions which are explained on the aforementioned website.

Regards,
Tobias

Legacy User

Checked it out with eclipse. Compile worked great, including the fix on the SOM preprocessing saving/loading issue. Thanks.