“occurre文本挖掘术语nces per label value"

arnoarno MemberPosts:4Contributor I
edited June 2019 inHelp
Hi everyone!

I started out using Rapidminer for text mining as it seems a pretty powerful tool to do so.

When using the"Process documents from data" operatorI get an output calledWordListwhich gives an overview of the different
words in the documents and a frequency of occurrence. I also set a label on the dataset and the table also shows the values
of this label as different categories for which it should give youterm occurrence frequencies. However while
"document occurences" and "Total occurence" seem to be calculated correctly for every word, all the different categories just show 0 for every word.

I would expect a word like let's say "sponsor" which occurs in 10 documents to be distributed over the different categories since every document was classified
in a category.

Did I do something wrong in the data import process? Are there prerequisites I do not know about so the division of word occurrences would be shown correctly over all the values of the label
variable?

thanks in advance,

Arno
Tagged:

Answers

  • SkirzynskiSkirzynski MemberPosts:164Maven
    Hey Arno,

    我不得到它或我不能reproduce this. Could you please provide a minimal example which can be reproduced, i.e. the process and a small set of data which will be loaded. You can use the code-tags to paste the XML of the process and the data in CSV format for instance.

    Cheers
    Marcin
  • arnoarno MemberPosts:4Contributor I
    This is the process flow in XML. it's actually nothing more than animportand somedata cleaning(tokenisation, stemming, stopword filtering and n-gram creation) inside a "Process documents from data" which generates theword list.

    How do I add data to a forum post?and a screenshot? cause i get the image tags but can't upload an image?:)

    Thedatais pretty simple though: just 1 variable filled with text and the other variable is a sentiment label (neutral, positive, negative).

    data is like this:

    content sentiment
    Limburg: Centrum Helchteren op de parking van de Carrefour zijn ze zich aan het opstellen neutral
    Limburg: Centrum Helchteren op de parking van de Carrefour zijn ze zich aan het opstellen neutral
    Ging goed het inwerken bij de Albert Heijn! Wel beetje chaotisch in het begin neutral
    STABROEK: DAAR MAG JE GOOIEN MET EIEREN!:p
     positive
    Limburg: Centrum Helchteren op de parking van de Carrefour zijn ze zich aan het... -@FlitscontroleBEneutral
    Al carrefour. Mi nenea te amo neutral
    @Beauux1moet je nog lidl? neutral
    RT@X_xAE: HahahahahahahhhH ik kom@Y0UKN0WITBR0. Altijd in van die coole winkels tegen hahahaha zeeman lidl allussss:)positive
    @sbfotos@hemaAlbert Heijn heeft een vergelijkbaar product! neutral
    whaha , jochem viel net in de lidl qqqq. hij lag zo mooi op de grondd, gehehe. positive
    @NeaKurt艾伯特gewoon bij heijn……窟denk我cvandaag met 35 procent korting.... neutral
    @CRAZYKiiiD__@catilin99ik ga gewoon naar appie (albert Heijn) in de stad of gewoon Deka of c1000 bij mij in de buurt neutral
    I'm at Delhaize (Schaarbeek / Schaerbeek, Brussels) http://4sq.com/11KYg1c" title="http://4sq.com/11KYg1c">4sq.com/11KYg1c neutral
    Redactie Foodlog: Is de bakker te dom of de Lidl te groot? -@foodlog_nl- http://bit.ly/UnUarr" title="http://bit.ly/UnUarr">bit.ly/UnUarr neutral
    Naar albert heijn daarna naar huis en eindelijk eten. neutral
    code:



































































  • SkirzynskiSkirzynski MemberPosts:164Maven
    Data in code-tags are just fine.:)

    The good news is that I could reproduce your issue. Even better, I know which operator is the reason for this. If you remove the "Extract Content" operator you will see correct values for the label. The bad news is that I am not sure why this happens and I am still investigating. It seems that it has something to do with some meta data this operator is adding to the document.

    As a workaround you can remove the HTML-tags without using the "Extract Content" operator. This can be done by using the "Replace" operator which you have to insert before the "Process Documents From Data" operator. Use
    <[/]?[^>]*>
    as the regular expression in the parameter "replace what" and leave "replace by" empty. This should remove all tags like the "Extract Content" operator did.

    I hope this helps
    Marcin
  • arnoarno MemberPosts:4Contributor I
    Thanks a lot for the feedback,

    I will test your solution soon and give you feedback on it once I have it;).

    thanks again,

    Arno
Sign InorRegisterto comment.