Loop operator issues

rur68

Hello, everyone
我想crawl a web with loop operator. the website defines first page is the latest pages. but the information i want to get is the first five page of the newest.
can loop operator in rapidminer do iteration backwards?

rfuentealba

Hello@rur68,

For the sake of simplicity, I'll be using this url instead of the one you provide:

http://jakarta.id/page/%{iteration}/site.

You are iterating numbers 1, 2 and 3. Right?

Well, with number 1, that URL becomes:

http://jakarta.id/page/1/site.

With number 2, that URL becomes:

http://jakarta.id/page/2/site.

With number 3, that URL becomes:

http://jakarta.id/page/1/site.

If you have a known page number (e.g., 500), then you may use theGenerate Macrooperator, giving a name to the macro generated (like: "calculated_page", with the following code:

500 + 1 - eval(%{iteration})

That way, with number 1 you will get:

http://jakarta.id/page/500/site.

With number 2, that URL becomes:

http://jakarta.id/page/499/site.

With number 3, that URL becomes:

http://jakarta.id/page/498/site.

However, that is for aknownnumber. If you are seeking for anunknownnumber, (e.g., some 1000 new results come every day and you want to crawl those), then you might be out of luck (but people in this community is amazing, they might come up with a solution) and I would recommend you to use something not-so-rapidminer-ish likehttrackon UNIX machines (Linux, Mac) to grab an updated copy of the site and use indexes or other tricks under the sleeve to handle these as files.

警告:httrackand other site crawlers might be prohibited in your country, your mileage may vary.

Hope this helps, if I can come up with a better solution, then I'll be back to this thread.

All the best,

Rodrigo.

rfuentealba

Great, glad it helped!

rfuentealba

Hello,

Are you able to share an example of your data, and your XML for us to see and answer properly? There are many ways to go backwards inside a loop operator, but I need to figure out what are you doing first.

All the best,

Rodrigo.

rur68

hello,@rfuentealba.

here is the example process. i want to get the newest three pages of the website, but the result of the process is the latest three pages.

rur68

yeah it works! thank you so much,@rfuentealba!!