Generate n-grams characters or terms operator?

ikayunida123

Hello everyone!

I'm still new to rapidminer and today I'm going to do a pre-processing for sentiment analysis using rapidminer :catvery-happy:

But I'm confused with the generate n-grams operator. There are two generate n-grams files in rapidminer, generate n-grams (character) and generate n-grams (terms). What's the difference between those two actually? I'm trying to figure it out with read the help section, but I'm still don't understand the difference.

I need your help. Thank you!

sgenzer

hello@ikayunida123- that is a good question. Have you readthe KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.

I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).

<宏/ >










<连接from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
<连接from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
<连接from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
<连接from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
<连接from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
<连接from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>

Does that help?

Scott

ikayunida123

It works. Thanks a lot!