You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Robert Irribarren <ro...@algorithms.io> on 2012/08/18 10:22:18 UTC
Solr Doesn't Add New Entries

After parsing lots of documents, I run
new procedure experiental
nutch inject urls
nutch generate
bin/nutch crawl urls -depth 3 -topN 10
bin/nutch solrindex http://127.0.0.1:8983/solr/ -reindex
echo Crawling completed
dir


I accept extern links in the nutch-site.xml

I have regex-urlfilter.txt to accept everything else

When I run readdb I get this result

WebTable statistics start
Statistics for WebTable:
min score:      0.0
status 2 (status_fetched):      1053
jobs:   {db_stats-job_local_0001={jobID=job_local_0001, jobName=db_stats,
counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce
Framework={MAP_OUTPUT_MATERIALIZED_BYTES=211, MAP_INPUT_RECORDS=1234,
REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=65418,
COMMITTED_HEAP_BYTES=504635392, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=1046,
COMBINE_INPUT_RECORDS=4936, REDUCE_INPUT_RECORDS=12,
REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=4936},
FileSystemCounters={FILE_BYTES_READ=878225, FILE_BYTES_WRITTEN=991145},
File Output Format Counters ={BYTES_WRITTEN=375}}}}
retry 0:        1233
retry 1:        1
TOTAL urls:     1234
status 4 (status_redir_temp):   32
status 5 (status_redir_perm):   47
max score:      1.0
status 34 (status_retry):       16
status 3 (status_gone): 17
status 0 (null):        69
avg score:      0.01614992
WebTable statistics: done
min score:      0.0
status 2 (status_fetched):      1053
jobs:   {db_stats-job_local_0001={jobID=job_local_0001, jobName=db_stats,
counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce
Framework={MAP_OUTPUT_MATERIALIZED_BYTES=211, MAP_INPUT_RECORDS=1234,
REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=65418,
COMMITTED_HEAP_BYTES=504635392, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=1046,
COMBINE_INPUT_RECORDS=4936, REDUCE_INPUT_RECORDS=12,
REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=4936},
FileSystemCounters={FILE_BYTES_READ=878225, FILE_BYTES_WRITTEN=991145},
File Output Format Counters ={BYTES_WRITTEN=375}}}}
retry 0:        1233
retry 1:        1
TOTAL urls:     1234
status 4 (status_redir_temp):   32
status 5 (status_redir_perm):   47
max score:      1.0
status 34 (status_retry):       16
status 3 (status_gone): 17
status 0 (null):        69
avg score:      0.01614992


I look at the database


I get 1234 entries into the webpage TABLE on my nutch database, but when i
run the solrindex -reindex I get
adding 36 documents
And I can't add any more new documents to solr. Please help