Feature Request: Model Simulator for Grouped Models

christos_karras

I have a "Grouped Model" that groups different model pre-processing steps, for example Normalization, Imputation, etc. I tried to use this model with the Model Simulator, but I get this error:
Process failed: Wrong input of type 'Grouped Model' at port 'model'. Expected type 'Model'.

As a workaround, I could re-apply all the pre-processing steps and use only the actual model with Model Simulator. However, this makes experimenting with different pre-processing steps (enable/disable normalization, enable/disable imputation, etc) harder because I have to duplicate code to make sure the steps are correctly applied for the model simulator.

Would it be feasible for a future version to support Grouped Models on the model simulator? Is there any technical reason why it wouldn't be supported, or is it just a missing feature?

MartinLiebig

Hi@christos_karras,

we are on it. Give us a bit.

Best,

Martin

CC:@IngoRMwho may reveal more

IngoRM

Now I can reveal a bit more :-)

We just have released a new Beta version of RM 9.6 and you can try this yourself:https://static.www.turtlecreekpls.com/rnd/html/rapidminer-9.6-preview.html

Cheers,

Ingo

christos_karras

Hi@IngoRM, Great, so I guess I had good timing to ask that question. However, I just tried beta 9.6 and I see that there are still cases of grouped models that don't work:

1. Nested grouped models: I have a first Group Model operator that is used for a sequence of pre-processing operations (normalization, PCA, etc). Then a second Group Model operator adds the prediction model. This results in a nested grouped model (a grouped model that contains another grouped model with the pre-processing steps). For this I have a workaround: I start by ungrouping the pre-processing model to get a collection of models, then I re-group this collection and the prediction model. This results in a flat group of models that works correctly.

2. Some pre-processing steps also cause the error. For example, if I have a PCA followed by Random Forest, I get an error saying that the last model must be a prediction model. With the same model but without the PCA, I don't get this error.

I tested this with the Model Simulator and Explain Predictions operators.

IngoRM

Hi@christos_karras

Thanks for the feedback - that is very helpful! Re 1) we probably will not support nested models, at least not in the next version. They are rarely nested and the workaround is relatively obvious (I hope). But let's monitor and see if more people run into the same issue...

(2)确实是一个意想不到的错误检查in the simulator is very rigorous and only allow preprocessing models which are defined as such, not general models which just behave like a preprocessing model. We will look into relaxing this check to make this work for all models.

Thanks again for pointing this out.
Best,
Ingo

IngoRM

Just FYI, I have created an internal ticket for the second point (IC-1764).

IngoRM

And another FYI: the issue has been resolved. Starting with RapidMiner 9.7, you can add any model to the grouped model, not just those which are (technically) preprocessing models. As a result, you can now also combine for example a PCA model with a prediciton model.
Hope this helps,
Ingo

christos_karras

Hi@IngoRM,

I tested this change successfully with RapidMiner 9.8 using the Model Simulator and the Explain Predictions operators. However, I tried to deploy this grouped model to model ops, and I still have a similar error:

Nov 17, 2020 4:48:57 PM com.rapidminer.extension.mdm.PluginInitModelManagement$3 done

WARNING: Problem during deployment in background thread: java.lang.IllegalArgumentException: Only prediction models or grouped models consisting of preprocessing models and ending with a prediction model are supported for deployments.

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Only prediction models or grouped models consisting of preprocessing models and ending with a prediction model are supported for deployments.

Could it simply be that a validation needs to be removed in Model Ops, now that Model Simulator and Explain Predictions support this? Or is there another limitation with Model Ops?

Thanks

IngoRM

Hi@christos_karras,

Thanks for letting us know. I just tried a couple of models including one and several preprocessing models with no problem. Can you share the set of (preprocessing models) and the chain of models you have been trying? I am happy to take a deeper look based on that.

Cheers,
Ingo

christos_karras

Hi@IngoRM,

I tried a "complex" example with:

- Normalization (Z-Transformation) for one subset of attributes

- Normalization (MinMaxNormalizationModel) for another subset of attributes

- PCA

- Final prediction model: Stacking

However, I get the same same error with something simpler, for example PCA or ICA followed by Random Forest.

I can create an example to reproduce the error later today if needed.

Thanks

christos_karras

Hi@IngoRM,

I created a sample process to generate grouped models that will cause this error. The process includes several models in the group to try to identify what works and what doesn't. It's probably more complex than a real grouped model would typically be, but I think it can be a good test case for various features that use grouped models.

This process will generate 2 files that can then be used to deploy a custom model to model ops:

/ /本地存储库/ ComplexGroupedModelData

//Local Repository/ComplexGroupedModel

The grouped model includes the following steps, and most of them work fine:

Only PCA and KMeans Clustering (used to generate a "cluster" attribute as a pre-processing step) cause the grouped model to fail to deploy to model ops, even though this grouped model works with Explain Predictions and Model Simulator (as I also test in the process)

轻松地允许测试删除PCA和KMeans Clustering resolves the issue, and adding either one of them causes the error, the process adds these two pre-processing models conditionally, based on two macros defined in the process context: IncludePCA and IncludeKMeans.

To reproduce the error:

* Set either IncludePCA or IncludeKMeans to 1, run the process, then deploy the resulting model to model ops

To create a model that can be deployed to model ops without error:

* Set both IncludePCA and IncludeKMeans to 0, run the process, then deploy the resulting model to model ops

IncludePCA
1


IncludeKMeans
1















<参数键=“attributes_lower_bound”值= -10.0"/>








<运营商激活= " true " class = " set_role”那么tibility="9.8.000" expanded="true" height="82" name="Set Role label (2)" width="90" x="447" y="187">





<参数键= =“false”/“create_nominal_ids”价值>


















<参数键= value =“use_value_type_exception false"/>



















<参数键= value =“use_value_type_exception false"/>



















<参数键= value =“use_value_type_exception false"/>







































<参数键= value =“use_value_type_exception false"/>




















<参数键= value =“use_value_type_exception false"/>















<连接from_port = " 1 " to_op =“离散化”att4to_port="example set input"/>















































<运营商激活= " true " class = " set_role”那么tibility="9.8.000" expanded="true" height="82" name="Set Role (2)" width="90" x="45" y="136">





























<运营商激活= " true " class = " set_role”那么tibility="9.8.000" expanded="true" height="82" name="Set Role (3)" width="90" x="313" y="34">












<参数键= value =“use_value_type_exception false"/>












<运营商激活= " true " class = " set_role”那么tibility="9.8.000" expanded="true" height="82" name="Set Role" width="90" x="782" y="34">



















































<运营商激活= " true " class = " set_role”那么tibility="9.8.000" expanded="true" height="82" name="Set Role label" width="90" x="1318" y="136">
























































































































































To debug the grouped models, we can ungroup them and apply them separately












































Test RapidMiner features that did not work with grouped models before v9.7