You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Sandeep Kulkarni <sa...@usc.edu> on 2015/09/26 03:52:18 UTC

Configuring rotating agent in Nutch

Hello,

I am trying to configure rotating agent in Nutch.

I have set following in nutch-site.xml but I cannot see different user ids
in crawl. What am I missing:

<property>
  <name>http.agent.rotate</name>
  <value>true</value>
  <description>
  </description>
</property>

<property>
  <name>http.agent.rotate.file</name>
  <value>agents.txt</value>
  <description>
  </description>
</property>

Thanks in advance for your help.

Re: Configuring rotating agent in Nutch

Posted by Karanjeet Singh <co...@gmail.com>.

I am facing the same problem here. Tried rebuilding it but in logs I can only
see the agent name mentioned in http.agent.name property.

By $NUTCH_HOME/conf do you mean runtime/local/conf directory ? 

Also can you please brief me on how the rotation works ? Does the agent
rotates after crawling some X links and if so can we configure that X ?



--
View this message in context: http://lucene.472066.n3.nabble.com/Configuring-rotating-agent-in-Nutch-tp4231459p4231609.html
Sent from the Nutch - User mailing list archive at Nabble.com.