operator for removal of emotions from twitter data

Arupriya_Sen

which operator am I supposed to use in order to remove emoticons or emotions from twitter data to conduct sentiment analysis? I use Rapidminer version 9.3.0

varunm1

Hello@Arupriya_Sen

I am not so sure, but I think tokenization removes these emoticons as they are represented in symbols with punctuations. Give it a try.

@kaymanor@sgenzerany suggestions here.

Thanks

sgenzer

I would convert the text to UTF-8 (use Encode URL operator) like this:

Tweet:

https://twitter.com/_harold17/status/1145676138561310720?s=20

Before encoding:



After encoding:


查找utf - 8的Unicode图表(如https://apps.timwhitlock.info/emoji/tables/unicode):



So 'face with tears of joy' emoji is %F0%9F%98%82 which makes sense as you see this from the encoded text:


and so on. So then it's just a matter of using Replace with %F0%9F%98%xx with the encoded text, then decode back. Something like this:

<？xml version = " 1.0 " encoding = " UTF -8"?>   
                                                              

Scott