how to tokenize documents?

kayman

I'm wondering if it's feasible to tokenize documents, as the tokenize operator itself only offers the option to tokenize on expressions and the likes.

As an example consider the following scenario : a collection of similar documents in a folder is loaded and combined in a single document using the combine document operator. Using the extract token number operator shows there are indeed n tokens in the document (where each token represents a loaded document) but there seems to be no option to loop through these tokens afterwards, or option to split again by token later in the process.

Is this indeed not possible or is there some cool but not so very visible option available that would allow me to tokenize on combined documents?

MartinLiebig

Hi@kayman,

now i understand. There is nothing to split on, but you want to export the tokens itself. Something like this groovy script i guess?

< ?xml version = " 1.0 " encoding = " utf - 8 " ?> <过程版本sion="9.3.001">
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Telcontar120

@kaymanthis is a great question---but I do not know of any way to tokenize on documents unless there is some consistent header information that would be able to be used in an expression.
@mschmitzis there another solution here?

MartinLiebig

Hi,
if i understand this correctly, you may have something like

这是一个文本;这是另一个text ; and here is yet another ;

and you want to break this up into separate documents, splitted on ;?

If yes, an easy way to do this would be the Split into Collection operator in Operator Toolbox.@tftemme's original use case was JSON parsing, so you get each item of an array as a document.

I think there are some ways of doing this with reguar regions in Cut Document as well.

Best,
Martin

kayman

Hi@mschmitz, not entirely. Splitting up on sentences would be fairly easy to do, but I am talking about 'full blown' documents that can contain multiple sentences and paragraphs so the typical line break / new line logic wouldn't work here.

When looking at the document result rapidminer shows the distinction between the documents (as the color is different from doc 2 doc) but there is no real way to get the actual tokens / separate docs. Unless there is this super secret special symbol used by rapidminer to distinguish where one doc (token) ends and another one starts.

MartinLiebig

Hey@kayman,

The Split into Collection operator has a setting to define the token. if you set it to your tokenize token it should work?

BR,

Martin

kayman

Hi@mschmitz, this still requires you to know what is actually separating the tokens, unless I am missing something.

Find attached simplified example :

< ?xml version = " 1.0 " encoding = " utf - 8 " ?> <过程版本sion="9.3.001">   
                                  

Just 2 no-sense documents stitched together for the exercise, but I want to split them again as document. The document outcome shows the two documents as separate tokens, but splitting them into actual docs isn't that straightforward it seems. I've tried a variety of regex expressions but they always provide me either single sentences, or nothing, but not the total document.

kayman

Thanks@mschmitz, not as out-of-the-box I was looking for but it does do the trick indeed.

MartinLiebig

@kaymani'll build the box, give me a 2-3 days

sgenzer

@kaymanso if I understand correctly, you're actually creating these documents by using Loop Files -> Read Document like this?


If so, I believe Loop Files actually inserts a space between docs (don't ask me why!) so you can just do this:

< ?xml version = " 1.0 " encoding = " utf - 8 " ?> <过程版本sion="9.3.001">   
                                                 

The folder I used for this is attached below so you can match my process. You'll have to change the path of Loop Files of course.

Scott

kayman

Thanks@sgenzer, though it seems to work for external docs it's a bit tricky if there would be new lines followed by a space in a given document. Happens quite a lot with html for instance so not really an option.

MartinLiebig

Hey@kayman,

i've created an operator for it in converters extension, which will hit marketplace as soon as@tftemmeapproves itThis also has an option to get the tokens as an ExampleSet.

BR,

Martin