You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by dogrdon <dg...@planning.org> on 2013/07/28 23:38:53 UTC
Deleting Duplicates works fine on one solr core, but not on
antother - Nutch 1.5
Hi all,
I am trying to figure out how to debug this situation, or figure out what I
am missing between two different cores. let me explain:
I started using Nutch to crawl and index our page to a solr core1. And it
worked fine. Job completed like it should.
Though I wanted to start indexing or page to our solr core0, along with
other items that we want to index.
The indexing is not the problem, it will crawl and index fine. But on core0
it continues to fail on the delete duplicates task at the end of the index.
I get the following error (below). From what I can tell, the schema.xml and
solrconfig.xml files have all the same things across core0 and core1 except
for in core0 the url field is no longer required as the other indexed items
(not the crawled web pages, but records from our db) don't have a url, so
the id field is the standard, required field across all of them. Could it be
this that's causing the problem? what is the deduper trying to do and what
is getting in its way? and how might I get this straightened out for core0?
thanks!:
The hadoop.log error:
-------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
2013-07-26 16:55:17,797 INFO solr.SolrIndexWriter - Indexing 157 documents
2013-07-26 16:55:30,407 INFO solr.SolrMappingReader - source: content dest:
content 2013-07-26 16:55:30,444 INFO solr.SolrMappingReader - source: title
dest: title 2013-07-26 16:55:30,444 INFO solr.SolrMappingReader - source:
host dest: host 2013-07-26 16:55:30,444 INFO solr.SolrMappingReader -
source: segment dest: segment 2013-07-26 16:55:30,444 INFO
solr.SolrMappingReader - source: boost dest: boost 2013-07-26 16:55:30,444
INFO solr.SolrMappingReader - source: digest dest: digest 2013-07-26
16:55:30,444 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
2013-07-26 16:55:30,444 INFO solr.SolrMappingReader - source: url dest: id
2013-07-26 16:55:30,444 INFO solr.SolrMappingReader - source: url dest: url
2013-07-26 16:55:31,590 INFO indexer.IndexingJob - Indexer: finished at
2013-07-26 16:55:31, elapsed: 00:00:19 2013-07-26 16:55:31,593 INFO
solr.SolrDeleteDuplicates - SolrDeleteDuplicates: starting at 2013-07-26
16:55:31 2013-07-26 16:55:31,593 INFO solr.SolrDeleteDuplicates -
SolrDeleteDuplicates: Solr url: http://<domain>:<port>/solr/core0/
2013-07-26 16:55:32,043 WARN mapred.FileOutputCommitter - Output path is
null in cleanup 2013-07-26 16:55:32,043 WARN mapred.LocalJobRunner -
job_local1142877999_0055 java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.lang.NullPointerException at
org.apache.hadoop.io.Text.encode(Text.java:388) at
org.apache.hadoop.io.Text.set(Text.java:178) at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:270)
at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:230)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:210)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:366) at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
java.util.concurrent.FutureTask.run(FutureTask.java:166) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
-------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/Deleting-Duplicates-works-fine-on-one-solr-core-but-not-on-antother-Nutch-1-5-tp4080931.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Deleting Duplicates works fine on one solr core, but not on
antother - Nutch 1.5
Posted by dogrdon <dg...@planning.org>.
I've brought up the logs particular to a recent crawl where the error
occurred, it is posted below. I don't see anything that really stands out.
On further consideration it seems as though on repeated crawls, it is
updating the webpage crawls and nothing is being duplicated. I think it is
choking when it looks at the other records in the index, because the nutch
crawls are being indexed against non-webpage entities.
Might it be that it is working fine, it's just delivering the error because
it can't dedupe things that did not originate from Nutch (and hence don't
have url fields associated with them?)
What's more, are there any problems indexing nutch records alongside
non-nutch records in a solr core? I am not really seeing any other than the
odd error message.
------------------------------------------------------------------------------------------------------
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
------------------------------------------------------------------------------------------------------
Jul 31, 2013 11:26:31 AM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/opt/solr/core0/data/index,segFN=segments_8r,version=1370615988315,generation=315,filenames=[_97.fdt,
_99.fdx, _98.tis, _99.nrm, _97.fnm, _98.tii, _99.prx, _97.fdx, _99.fdt,
_99.frq, _96.prx, _96.frq, _98.nrm, _96.fnm, _97.frq, segments_8r, _99.tis,
_96.tii, _99.tii, _96.tis, _99.fnm, _96.fdx, _98.fdt, _97.tis, _97.nrm,
_96.nrm, _98.fnm, _97.prx, _96.fdt, _98.prx, _97.tii, _98.fdx, _98.frq]
Jul 31, 2013 11:26:31 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1370615988315
Jul 31, 2013 11:26:46 AM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[http://www.oursite.com/, http://www.oursite.com/about/,
http://www.oursite.com/aboutus/,
http://www.oursite.com/aboutus/becomingascanner.htm,
http://www.oursite.com/aboutus/whatisscanning.htm,
http://www.oursite.com/aboutus/yourspace.htm,
http://www.oursite.com/advertise/, http://www.oursite.com/advocacy/, ...
(158 adds)]} 0 15127
Jul 31, 2013 11:26:46 AM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=/solr path=/update params={wt=javabin&version=2}
status=0 QTime=15127
Jul 31, 2013 11:26:46 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=/opt/solr/core0/data/index,segFN=segments_8r,version=1370615988315,generation=315,filenames=[_97.fdt,
_99.fdx, _98.tis, _99.nrm, _97.fnm, _98.tii, _99.prx, _97.fdx, _99.fdt,
_99.frq, _96.prx, _96.frq, _98.nrm, _96.fnm, _97.frq, segments_8r, _99.tis,
_96.tii, _99.tii, _96.tis, _99.fnm, _96.fdx, _98.fdt, _97.tis, _97.nrm,
_96.nrm, _98.fnm, _97.prx, _96.fdt, _98.prx, _97.tii, _98.fdx, _98.frq]
commit{dir=/opt/solr/core0/data/index,segFN=segments_8s,version=1370615988320,generation=316,filenames=[_97.fdt,
_98.tis, _99.nrm, _97.fnm, _98.tii, _9b.tii, _9a.tii, _97.fdx, _9a.tis,
_96.prx, _96.frq, _96.fnm, _9b.tis, _97.frq, _9a.prx, _9a.fdx, segments_8s,
_9b.nrm, _96.tii, _96.tis, _96.fdx, _9b.fdx, _98.fdt, _9a.nrm, _9b.fdt,
_98.fnm, _98.prx, _96.fdt, _98.fdx, _98.frq, _9a.fdt, _99.fdx, _9b.fnm,
_99.prx, _99.fdt, _99.frq, _98.nrm, _99.tis, _9a.fnm, _98_1.del, _99.tii,
_99_1.del, _9a.frq, _99.fnm, _9b.prx, _97.nrm, _97.tis, _96.nrm, _9b.frq,
_97.prx, _97.tii]
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1370615988320
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
queryResultCache{lookups=68,hits=56,hitratio=0.82,inserts=13,evictions=0,size=13,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
documentCache{lookups=3040,hits=2992,hitratio=0.98,inserts=50,evictions=0,size=50,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jul 31, 2013 11:26:49 AM org.apache.solr.core.QuerySenderListener
newSearcher
INFO: QuerySenderListener sending requests to Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.core.QuerySenderListener
newSearcher
INFO: QuerySenderListener done.
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrCore registerSearcher
INFO: [core0] Registered new searcher Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@5484ff20 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
queryResultCache{lookups=68,hits=56,hitratio=0.82,inserts=13,evictions=0,size=13,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
documentCache{lookups=3040,hits=2992,hitratio=0.98,inserts=50,evictions=0,size=50,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.update.processor.LogUpdateProcessor
finish
------------------------------------------------------------------------------------------------------
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
------------------------------------------------------------------------------------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/Deleting-Duplicates-works-fine-on-one-solr-core-but-not-on-antother-Nutch-1-5-tp4080931p4081671.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Deleting Duplicates works fine on one solr core, but not on
antother - Nutch 1.5
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Makes perfect sense.
I wonder if this is something to do with the Solr side?
Do you have some logs you can view?
On Tue, Jul 30, 2013 at 9:48 AM, dogrdon <dg...@planning.org> wrote:
> oh, sorry, I just obscured it for the purposes of posting because I did not
> want to publish our solr installation to the rest of the internet
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Deleting-Duplicates-works-fine-on-one-solr-core-but-not-on-antother-Nutch-1-5-tp4080931p4081356.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
*Lewis*
Re: Deleting Duplicates works fine on one solr core, but not on
antother - Nutch 1.5
Posted by dogrdon <dg...@planning.org>.
oh, sorry, I just obscured it for the purposes of posting because I did not
want to publish our solr installation to the rest of the internet
--
View this message in context: http://lucene.472066.n3.nabble.com/Deleting-Duplicates-works-fine-on-one-solr-core-but-not-on-antother-Nutch-1-5-tp4080931p4081356.html
Sent from the Nutch - User mailing list archive at Nabble.com.