You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by 基勇 <25...@qq.com> on 2014/05/14 10:26:23 UTC
using solr indexing exception
Hi!gays
I run the following command in nutch2.2.1,
bin/nutch crawl urls/ -solr http://localhost:8983/solr -depth 3 -topN 20
Exception in thread "main" java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
at org.apache.nutch.indexer.IndexerJob.createIndexJob(IndexerJob.java:129)
at org.apache.nutch.indexer.solr.SolrIndexerJob.run(SolrIndexerJob.java:44)
at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:69)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:194)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:252)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawler.main(Crawler.java:261)
the nutch-site.xml file：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>test-nutch</value>
</property>
<property>
<name>http.robots.agents</name>
<value>test-nutch,*</value>
</property>
<property>
<name>generate.batch.id</name>
<value>1</value>
</property>
<property>
<name>http.agent.name.check</name>
<value>true</value>
</property>
<property>
<name>distributed.search.test.port</name>
<value>60000</value>
</property>
<property>
<name>http.accept.language</name>
<value>ja-jp,en-us,en-gb,en;q=0.7,*;q=0.3</value>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hduser/dirs/hdp_tmp_dir</value>
</property>
<property>
<name>plugin.folders</name>
<value>/home/hduser/apache-nutch-2.2.1/runtime/local/plugins</value>
</property>
</configuration>
I don't know how to fix it.
The source code :
protected Job createIndexJob(Configuration conf, String jobName, String batchId)
127
throws IOException, ClassNotFoundException {
128
conf.set(GeneratorJob.BATCH_ID, batchId);
BATCH_ID
public static final String BATCH_ID = "generate.batch.id";
How to fix it.I need you help!
Thank you !
Re: 回复： using solr indexing exception
Posted by feng lu <am...@gmail.com>.
It seems that you don't set batch id correctly. I see Crawler class is not
used in launch script so you can try bin/nutch or bin/crawl command to run
the Nutch again.
On Thu, May 15, 2014 at 9:10 AM, 基勇 <25...@qq.com> wrote:
> Which friend can help solve this problem?
> Thank's
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "真心の傻子楓";<25...@qq.com>;
> 发送时间: 2014年5月14日(星期三) 下午4:26
> 收件人: "user"<us...@nutch.apache.org>;
>
> 主题: using solr indexing exception
>
>
>
> Hi!gays
>
> I run the following command in nutch2.2.1,
> bin/nutch crawl urls/ -solr http://localhost:8983/solr -depth 3 -topN 20
>
> Exception in thread "main" java.lang.NullPointerException
> at java.util.Hashtable.put(Hashtable.java:394)
> at java.util.Properties.setProperty(Properties.java:143)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
> at org.apache.nutch.indexer.IndexerJob.createIndexJob(IndexerJob.java:129)
> at org.apache.nutch.indexer.solr.SolrIndexerJob.run(SolrIndexerJob.java:44)
> at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:69)
> at org.apache.nutch.crawl.Crawler.run(Crawler.java:194)
> at org.apache.nutch.crawl.Crawler.run(Crawler.java:252)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawler.main(Crawler.java:261)
>
> the nutch-site.xml file：
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
>
>
> <!-- Put site-specific property overrides in this file. -->
>
>
>
>
> <configuration>
>
> <property>
>
> <name>http.agent.name</name>
>
> <value>test-nutch</value>
>
> </property>
>
> <property>
>
> <name>http.robots.agents</name>
>
> <value>test-nutch,*</value>
>
> </property>
>
> <property>
>
> <name>generate.batch.id</name>
>
> <value>1</value>
>
> </property>
>
> <property>
>
> <name>http.agent.name.check</name>
>
> <value>true</value>
>
> </property>
>
> <property>
>
> <name>distributed.search.test.port</name>
>
> <value>60000</value>
>
> </property>
>
> <property>
>
> <name>http.accept.language</name>
>
> <value>ja-jp,en-us,en-gb,en;q=0.7,*;q=0.3</value>
>
> </property>
>
> <property>
>
> <name>parser.character.encoding.default</name>
>
> <value>utf-8</value>
>
> </property>
>
> <property>
>
> <name>storage.data.store.class</name>
>
> <value>org.apache.gora.hbase.store.HBaseStore</value>
>
> </property>
>
> <property>
>
> <name>hadoop.tmp.dir</name>
>
> <value>/home/hduser/dirs/hdp_tmp_dir</value>
>
> </property>
>
> <property>
>
> <name>plugin.folders</name>
>
> <value>/home/hduser/apache-nutch-2.2.1/runtime/local/plugins</value>
>
> </property>
>
> </configuration>
>
> I don't know how to fix it.
>
> The source code :
> protected Job createIndexJob(Configuration conf, String jobName, String
> batchId)
>
> 127
>
> throws IOException, ClassNotFoundException {
>
> 128
>
> conf.set(GeneratorJob.BATCH_ID, batchId);
>
>
> BATCH_ID
> public static final String BATCH_ID = "generate.batch.id";
>
>
> How to fix it.I need you help!
> Thank you !
>
--
Don't Grow Old, Grow Up... :-)
回复： using solr indexing exception
Posted by 基勇 <25...@qq.com>.
Which friend can help solve this problem?
Thank's
------------------ 原始邮件 ------------------
发件人: "真心の傻子楓";<25...@qq.com>;
发送时间: 2014年5月14日(星期三) 下午4:26
收件人: "user"<us...@nutch.apache.org>;
主题: using solr indexing exception
Hi!gays
I run the following command in nutch2.2.1,
bin/nutch crawl urls/ -solr http://localhost:8983/solr -depth 3 -topN 20
Exception in thread "main" java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
at org.apache.nutch.indexer.IndexerJob.createIndexJob(IndexerJob.java:129)
at org.apache.nutch.indexer.solr.SolrIndexerJob.run(SolrIndexerJob.java:44)
at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:69)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:194)
at org.apache.nutch.crawl.Crawler.run(Crawler.java:252)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawler.main(Crawler.java:261)
the nutch-site.xml file：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>test-nutch</value>
</property>
<property>
<name>http.robots.agents</name>
<value>test-nutch,*</value>
</property>
<property>
<name>generate.batch.id</name>
<value>1</value>
</property>
<property>
<name>http.agent.name.check</name>
<value>true</value>
</property>
<property>
<name>distributed.search.test.port</name>
<value>60000</value>
</property>
<property>
<name>http.accept.language</name>
<value>ja-jp,en-us,en-gb,en;q=0.7,*;q=0.3</value>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hduser/dirs/hdp_tmp_dir</value>
</property>
<property>
<name>plugin.folders</name>
<value>/home/hduser/apache-nutch-2.2.1/runtime/local/plugins</value>
</property>
</configuration>
I don't know how to fix it.
The source code :
protected Job createIndexJob(Configuration conf, String jobName, String batchId)
127
throws IOException, ClassNotFoundException {
128
conf.set(GeneratorJob.BATCH_ID, batchId);
BATCH_ID
public static final String BATCH_ID = "generate.batch.id";
How to fix it.I need you help!
Thank you !