Include Negations in Dictionary based Sentiment Approach

katrin_kandlbin

Hi all,

I want to use the dictionary based sentiment approach for texts by using a self-created dictionary.

The dictionary sentiment approach does not account for negations as far as I know.

For example: Today I wasnotreallyproductive.

The word "productive" ispositivebut because of the "not" the whole sentence has anegativemeaning instead of positive.

How does the dictionary based sentiment approach of Rapidminer account for this issue?

All the best,

Katrin

MartinLiebig

Hi Katrin,

welcome to the community! I've written the operator myself and I am very happy that you use it. The current version of the operator does not handle negations. Negations are in general a tricky thing also in more sophisticated approaches like SVMs. I am not sure how this dictionary based approach could handle this. If you have a solution in mind I am happy to have a look if we can implement this.

Cheers,

Martin

DocMusher

Hi,

Negation is in fact only a small part of the problem: imagine the sentence "probably a possible tumor" in the notes of a radiologist.

Sven

katrin_kandlbin

Hi Martin,

first of all: happy new year. Thank you very much for your reply.

One possible solution to account for negations might be to include another "dictionary" of self-created word list which consists of negation words.

This word list is applied to the same text corpus as the sentiment dictionary in the following way:

1. all words in the text corpus should be numberedconsecutively
2. the sentiment dictionary operator recognizes that (for example) word Nr. 12 is a sentiment word and assigns the corresponding value (1 if positive and -1 if negative)
3. the new negation operator then concentrates only on the sentiment words of the text corpus (here word Nr. 12) and examines the surrounding words - let's say 5 words before the sentiment word and 5 words after.
4. If there is a negation word included 5 words before or after the sentiment word, the value of the sentiment word will be multiplied by (-1) in order to reverse the sentiment value.

Do you think this might work?

Best,

Katrin

MartinLiebig

Hi@katrin_kandlbin,

i've implemented this (without counting but same idea). I will share a preview version of this via e-mail since i need to write some docs and it needs to run through our internal code-quality process before it appears on market place.

Best,

Martin