Crawl Web user agent is always Java/1.7_11

n0n0n0n0 MemberPosts:1Contributor I
edited October 2019 inHelp
Hello to the community.

Trying to crawl a website, i always get the mobile version of this site.
I thougth about an issue with the user agent, so I changed the user agent parameter in the Crawl Web process, but still get the same result.
I then tried to crawl the pagehttp://whatsmyuseragent.com/with the following parameters:





<宏/ >











http://whatsmyuseragent.com/"/>





















and the C:\Temp\0.html file i get reports my user agent to be Java/1.7.0_11, and this whatever i set in the user agent field..

I'm on a Windows 8 x64 machine, using RapidMiner 5.3.008 and Web Mining Extension 5.3.0

Any advice?
Thank you
n0n0
Tagged:

Answers

  • SkirzynskiSkirzynski MemberPosts:164Maven
    Hey,

    Seems to be a bug. For the "Get Page" operator it is working, but for "Crawl Web" and "Process documents from Web" not. I have created a ticket for this. We will come back to this thread once we have fixed this.

    Thank you for reporting
    Marcin
Sign InorRegisterto comment.