[jira] [Created] (NUTCH-1571) SolrInputSplit doesn't implement Writable and crawl script doesn't pass crawlId to generate and updatedb tasks

["yuanyun.cn (JIRA)" <ji...@apache.org>]

yuanyun.cn created NUTCH-1571:
---------------------------------

             Summary: SolrInputSplit doesn't implement Writable and crawl script doesn't pass crawlId to generate and updatedb tasks
                 Key: NUTCH-1571
                 URL: https://issues.apache.org/jira/browse/NUTCH-1571
             Project: Nutch
          Issue Type: Bug
          Components: indexer
    Affects Versions: 2.1
            Reporter: yuanyun.cn
             Fix For: 2.2


I met tow issues when I run crawl script from 2.x brunch.
1. It throws exception when run solrdedup task:
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
        at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeNewSplits(JobSplitWriter.java:123)
        at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:74)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:968)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
        at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
        at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:371)
        at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.run(SolrDeleteDuplicates.java:381)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.main(SolrDeleteDuplicates.java:391)

Debugged the code, and found this is because SolrInputSplit class doesn't implement Writable interface, so I made the change to implement the interface and the two methods: readFields and write.

2. Nothing is really pushed to remote solr server. Looked at the code, and found out this is because tasks: generate and updatedb doesn't use crawlId parameter: added "-crawlId $CRAWL_ID" to them and crawl script works well now.

Also seems generate task doesn't use paramters: $CRAWL_ID/crawldb $CRAWL_ID/segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira