You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by micklai <la...@gmail.com> on 2013/04/20 21:00:50 UTC
[Exception in thread "main" java.io.IOException: Job failed!]
Hi,
Env:
System ubuntu 12.04
Tomcat: 7.0.39
Solr: 3.6.2
Nutch: 1.6
Try to deploy nutch 1.6 with solr 3.6.2 but failed with running the command
below:
bin/nutch crawl urls -solr http://localhost:8080/solr/ -dir crawl -depth 2
-threads 5 -topN 5
For the details:
=============================================
crawl started in: crawl
rootUrlDir = urls
threads = 5
depth = 2
solrUrl=http://localhost:8080/solr/
topN = 5
Injector: starting at 2013-04-21 02:21:12
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-04-21 02:21:27, elapsed: 00:00:14
Generator: starting at 2013-04-21 02:21:27
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20130421022135
Generator: finished at 2013-04-21 02:21:42, elapsed: 00:00:15
Fetcher: Your 'http.agent.name' value should be listed first in
'http.robots.agents' property.
Fetcher: starting at 2013-04-21 02:21:42
Fetcher: segment: crawl/segments/20130421022135
Using queue mode : byHost
Fetcher: threads: 5
Fetcher: time-out divisor: 2
QueueFeeder finished: total 1 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetching http://www.163.com/
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-04-21 02:21:49, elapsed: 00:00:07
ParseSegment: starting at 2013-04-21 02:21:49
ParseSegment: segment: crawl/segments/20130421022135
Parsed (24ms):http://www.163.com/
ParseSegment: finished at 2013-04-21 02:21:56, elapsed: 00:00:07
CrawlDb update: starting at 2013-04-21 02:21:56
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20130421022135]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-04-21 02:22:09, elapsed: 00:00:13
Generator: starting at 2013-04-21 02:22:09
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20130421022217
Generator: finished at 2013-04-21 02:22:25, elapsed: 00:00:15
Fetcher: Your 'http.agent.name' value should be listed first in
'http.robots.agents' property.
Fetcher: starting at 2013-04-21 02:22:25
Fetcher: segment: crawl/segments/20130421022217
Using queue mode : byHost
Fetcher: threads: 5
Fetcher: time-out divisor: 2
QueueFeeder finished: total 5 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
fetching http://m.163.com/
Using queue mode : byHost
Using queue mode : byHost
fetching http://3g.163.com/links/4145
fetching http://music.163.com/
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetching http://caipiao.163.com/mobile/client_cp.jsp
-activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://m.163.com
maxThreads = 1
inProgress = 0
crawlDelay = 5000
minCrawlDelay = 0
nextFetchTime = 1366482150482
now = 1366482146346
0. http://m.163.com/newsapp/
-activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://m.163.com
maxThreads = 1
inProgress = 0
crawlDelay = 5000
minCrawlDelay = 0
nextFetchTime = 1366482150482
now = 1366482147348
0. http://m.163.com/newsapp/
-activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://m.163.com
maxThreads = 1
inProgress = 0
crawlDelay = 5000
minCrawlDelay = 0
nextFetchTime = 1366482150482
now = 1366482148350
0. http://m.163.com/newsapp/
-activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://m.163.com
maxThreads = 1
inProgress = 0
crawlDelay = 5000
minCrawlDelay = 0
nextFetchTime = 1366482150482
now = 1366482149352
0. http://m.163.com/newsapp/
-activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://m.163.com
maxThreads = 1
inProgress = 0
crawlDelay = 5000
minCrawlDelay = 0
nextFetchTime = 1366482150482
now = 1366482150354
0. http://m.163.com/newsapp/
fetching http://m.163.com/newsapp/
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-04-21 02:22:38, elapsed: 00:00:13
ParseSegment: starting at 2013-04-21 02:22:38
ParseSegment: segment: crawl/segments/20130421022217
Parsed (4ms):http://caipiao.163.com/mobile/client_cp.jsp
Parsed (3ms):http://m.163.com/newsapp/
Parsed (1ms):http://music.163.com/
ParseSegment: finished at 2013-04-21 02:22:45, elapsed: 00:00:07
CrawlDb update: starting at 2013-04-21 02:22:45
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20130421022217]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
*CrawlDb update: 404 purging: false*
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-04-21 02:22:58, elapsed: 00:00:13
LinkDb: starting at 2013-04-21 02:22:58
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: internal links will be ignored.
LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421022135
LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421022217
LinkDb: finished at 2013-04-21 02:23:08, elapsed: 00:00:10
SolrIndexer: starting at 2013-04-21 02:23:08
*SolrIndexer: deleting gone documents: false
SolrIndexer: URL filtering: false
SolrIndexer: URL normalizing: false*
Indexing 4 documents
*java.io.IOException: Job failed!*
SolrDeleteDuplicates: starting at 2013-04-21 02:23:39
SolrDeleteDuplicates: Solr url: http://localhost:8080/solr/
*Exception in thread "main" java.io.IOException: Job failed!*
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
=============================================
But nutch works well, it works with the command below:
bin/nutch crawl urls -dir crawl -depth 2 -topN 5
And I also had modified the [Solr_Home]/conf/schema.xml to make nutch
integrated with solr, solr also works well by accessing
"localhost:8080/solr".
Hope you can help in this problem, wait for your reply.
Thanks.
Br,
Mick
--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by kiran chitturi <ch...@gmail.com>.
Hi Mick,
You should see the logs for more information. They are present in
'logs/hadoop.log'
On Sat, Apr 20, 2013 at 3:00 PM, micklai <la...@gmail.com> wrote:
> Hi,
>
> Env:
> System ubuntu 12.04
> Tomcat: 7.0.39
> Solr: 3.6.2
> Nutch: 1.6
>
> Try to deploy nutch 1.6 with solr 3.6.2 but failed with running the command
> below:
> bin/nutch crawl urls -solr http://localhost:8080/solr/ -dir crawl -depth 2
> -threads 5 -topN 5
>
> For the details:
> =============================================
> crawl started in: crawl
> rootUrlDir = urls
> threads = 5
> depth = 2
> solrUrl=http://localhost:8080/solr/
> topN = 5
> Injector: starting at 2013-04-21 02:21:12
> Injector: crawlDb: crawl/crawldb
> Injector: urlDir: urls
> Injector: Converting injected urls to crawl db entries.
> Injector: total number of urls rejected by filters: 0
> Injector: total number of urls injected after normalization and filtering:
> 1
> Injector: Merging injected urls into crawl db.
> Injector: finished at 2013-04-21 02:21:27, elapsed: 00:00:14
> Generator: starting at 2013-04-21 02:21:27
> Generator: Selecting best-scoring urls due for fetch.
> Generator: filtering: true
> Generator: normalizing: true
> Generator: topN: 5
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls for politeness.
> Generator: segment: crawl/segments/20130421022135
> Generator: finished at 2013-04-21 02:21:42, elapsed: 00:00:15
> Fetcher: Your 'http.agent.name' value should be listed first in
> 'http.robots.agents' property.
> Fetcher: starting at 2013-04-21 02:21:42
> Fetcher: segment: crawl/segments/20130421022135
> Using queue mode : byHost
> Fetcher: threads: 5
> Fetcher: time-out divisor: 2
> QueueFeeder finished: total 1 records + hit by time limit :0
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Fetcher: throughput threshold: -1
> Fetcher: throughput threshold retries: 5
> fetching http://www.163.com/
> -finishing thread FetcherThread, activeThreads=4
> -finishing thread FetcherThread, activeThreads=3
> -finishing thread FetcherThread, activeThreads=2
> -finishing thread FetcherThread, activeThreads=1
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> -finishing thread FetcherThread, activeThreads=0
> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> -activeThreads=0
> Fetcher: finished at 2013-04-21 02:21:49, elapsed: 00:00:07
> ParseSegment: starting at 2013-04-21 02:21:49
> ParseSegment: segment: crawl/segments/20130421022135
> Parsed (24ms):http://www.163.com/
> ParseSegment: finished at 2013-04-21 02:21:56, elapsed: 00:00:07
> CrawlDb update: starting at 2013-04-21 02:21:56
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20130421022135]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> CrawlDb update: 404 purging: false
> CrawlDb update: Merging segment data into db.
> CrawlDb update: finished at 2013-04-21 02:22:09, elapsed: 00:00:13
> Generator: starting at 2013-04-21 02:22:09
> Generator: Selecting best-scoring urls due for fetch.
> Generator: filtering: true
> Generator: normalizing: true
> Generator: topN: 5
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls for politeness.
> Generator: segment: crawl/segments/20130421022217
> Generator: finished at 2013-04-21 02:22:25, elapsed: 00:00:15
> Fetcher: Your 'http.agent.name' value should be listed first in
> 'http.robots.agents' property.
> Fetcher: starting at 2013-04-21 02:22:25
> Fetcher: segment: crawl/segments/20130421022217
> Using queue mode : byHost
> Fetcher: threads: 5
> Fetcher: time-out divisor: 2
> QueueFeeder finished: total 5 records + hit by time limit :0
> Using queue mode : byHost
> Using queue mode : byHost
> fetching http://m.163.com/
> Using queue mode : byHost
> Using queue mode : byHost
> fetching http://3g.163.com/links/4145
> fetching http://music.163.com/
> Using queue mode : byHost
> Fetcher: throughput threshold: -1
> Fetcher: throughput threshold retries: 5
> fetching http://caipiao.163.com/mobile/client_cp.jsp
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482146346
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482147348
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482148350
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482149352
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482150354
> 0. http://m.163.com/newsapp/
> fetching http://m.163.com/newsapp/
> -finishing thread FetcherThread, activeThreads=4
> -finishing thread FetcherThread, activeThreads=3
> -finishing thread FetcherThread, activeThreads=2
> -finishing thread FetcherThread, activeThreads=1
> -finishing thread FetcherThread, activeThreads=0
> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> -activeThreads=0
> Fetcher: finished at 2013-04-21 02:22:38, elapsed: 00:00:13
> ParseSegment: starting at 2013-04-21 02:22:38
> ParseSegment: segment: crawl/segments/20130421022217
> Parsed (4ms):http://caipiao.163.com/mobile/client_cp.jsp
> Parsed (3ms):http://m.163.com/newsapp/
> Parsed (1ms):http://music.163.com/
> ParseSegment: finished at 2013-04-21 02:22:45, elapsed: 00:00:07
> CrawlDb update: starting at 2013-04-21 02:22:45
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20130421022217]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> *CrawlDb update: 404 purging: false*
> CrawlDb update: Merging segment data into db.
> CrawlDb update: finished at 2013-04-21 02:22:58, elapsed: 00:00:13
> LinkDb: starting at 2013-04-21 02:22:58
> LinkDb: linkdb: crawl/linkdb
> LinkDb: URL normalize: true
> LinkDb: URL filter: true
> LinkDb: internal links will be ignored.
> LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022135
> LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022217
> LinkDb: finished at 2013-04-21 02:23:08, elapsed: 00:00:10
> SolrIndexer: starting at 2013-04-21 02:23:08
> *SolrIndexer: deleting gone documents: false
> SolrIndexer: URL filtering: false
> SolrIndexer: URL normalizing: false*
> Indexing 4 documents
> *java.io.IOException: Job failed!*
> SolrDeleteDuplicates: starting at 2013-04-21 02:23:39
> SolrDeleteDuplicates: Solr url: http://localhost:8080/solr/
> *Exception in thread "main" java.io.IOException: Job failed!*
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
> at
>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
> at
>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> =============================================
> But nutch works well, it works with the command below:
> bin/nutch crawl urls -dir crawl -depth 2 -topN 5
>
> And I also had modified the [Solr_Home]/conf/schema.xml to make nutch
> integrated with solr, solr also works well by accessing
> "localhost:8080/solr".
>
> Hope you can help in this problem, wait for your reply.
> Thanks.
>
> Br,
> Mick
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
Kiran Chitturi
<http://www.linkedin.com/in/kiranchitturi>
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by micklai <la...@gmail.com>.
Hi,
I rebuild all the project as NutchTutorial, it works now!!!
Thank you for your kindly support, thought I don't know why it work...I'll
still check this and hope we can have further discussion about Nutch
development.
Br,
Mick
--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620p4058328.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by Lewis John Mcgibbney <le...@gmail.com>.
logging explicitly states that no solrUrl is set.
On Sunday, April 21, 2013, kiran chitturi <ch...@gmail.com> wrote:
> Hi Mick,
>
> Since this is an error with Indexing, Can you check the logs from Solr
side
> ?
>
>
> On Sun, Apr 21, 2013 at 4:15 AM, micklai <la...@gmail.com> wrote:
>
>> HI Kiran,
>>
>> Thank you for you support, but since practiced by following
NutchTutorial,
>> it still failed, please refer to the hadoop.log below....
>> 2013-04-21 02:51:50,025 WARN crawl.Crawl - solrUrl is not set, indexing
>> will be skipped...
>> 2013-04-21 02:51:50,679 INFO crawl.Crawl - crawl started in: crawl
>> 2013-04-21 02:51:50,679 INFO crawl.Crawl - rootUrlDir = urls
>> 2013-04-21 02:51:50,680 INFO crawl.Crawl - threads = 10
>> 2013-04-21 02:51:50,680 INFO crawl.Crawl - depth = 2
>> 2013-04-21 02:51:50,680 INFO crawl.Crawl - solrUrl=null
>> 2013-04-21 02:51:50,680 INFO crawl.Crawl - topN = 5
>> 2013-04-21 02:51:50,765 INFO crawl.Injector - Injector: starting at
>> 2013-04-21 02:51:50
>> 2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: crawlDb:
>> crawl/crawldb
>> 2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: urlDir: urls
>> 2013-04-21 02:51:50,848 INFO crawl.Injector - Injector: Converting
>> injected
>> urls to crawl db entries.
>> 2013-04-21 02:51:51,158 WARN util.NativeCodeLoader - Unable to load
>> native-hadoop library for your platform... using builtin-java classes
where
>> applicable
>> 2013-04-21 02:51:51,331 WARN snappy.LoadSnappy - Snappy native library
not
>> loaded
>> 2013-04-21 02:51:52,290 INFO plugin.PluginRepository - Plugins: looking
>> in:
>> /home/lailx/search_engine/nutch/plugins
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Plugin
>> Auto-activation mode: [true]
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Registered
Plugins:
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - the nutch
>> core
>> extension points (nutch-extensionpoints)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Regex URL
>> Normalizer (urlnormalizer-regex)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - CyberNeko
>> HTML
>> Parser (lib-nekohtml)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - OPIC
>> Scoring
>> Plug-in (scoring-opic)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic URL
>> Normalizer (urlnormalizer-basic)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Tika
>> Parser Plug-in
>> (parse-tika)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Html
Parse
>> Plug-in
>> (parse-html)
>> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic
>> Indexing
>> Filter (index-basic)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Anchor
>> Indexing
>> Filter (index-anchor)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - HTTP
>> Framework
>> (lib-http)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL
>> Filter
>> (urlfilter-regex)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL
>> Filter
>> Framework (lib-regex-filter)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository -
>> Pass-through URL
>> Normalizer (urlnormalizer-pass)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Http
>> Protocol
>> Plug-in (protocol-http)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Registered
>> Extension-Points:
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL
>> Normalizer (org.apache.nutch.net.URLNormalizer)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
>> Protocol
>> (org.apache.nutch.protocol.Protocol)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
>> Segment Merge
>> Filter (org.apache.nutch.segment.SegmentMergeFilter)
>> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL
>> Filter
>> (org.apache.nutch.net.URLFilter)
> --
> Kiran Chitturi
>
> <http://www.linkedin.com/in/kiranchitturi>
>
--
*Lewis*
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by kiran chitturi <ch...@gmail.com>.
Hi Mick,
Since this is an error with Indexing, Can you check the logs from Solr side
?
On Sun, Apr 21, 2013 at 4:15 AM, micklai <la...@gmail.com> wrote:
> HI Kiran,
>
> Thank you for you support, but since practiced by following NutchTutorial,
> it still failed, please refer to the hadoop.log below....
> 2013-04-21 02:51:50,025 WARN crawl.Crawl - solrUrl is not set, indexing
> will be skipped...
> 2013-04-21 02:51:50,679 INFO crawl.Crawl - crawl started in: crawl
> 2013-04-21 02:51:50,679 INFO crawl.Crawl - rootUrlDir = urls
> 2013-04-21 02:51:50,680 INFO crawl.Crawl - threads = 10
> 2013-04-21 02:51:50,680 INFO crawl.Crawl - depth = 2
> 2013-04-21 02:51:50,680 INFO crawl.Crawl - solrUrl=null
> 2013-04-21 02:51:50,680 INFO crawl.Crawl - topN = 5
> 2013-04-21 02:51:50,765 INFO crawl.Injector - Injector: starting at
> 2013-04-21 02:51:50
> 2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: crawlDb:
> crawl/crawldb
> 2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: urlDir: urls
> 2013-04-21 02:51:50,848 INFO crawl.Injector - Injector: Converting
> injected
> urls to crawl db entries.
> 2013-04-21 02:51:51,158 WARN util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2013-04-21 02:51:51,331 WARN snappy.LoadSnappy - Snappy native library not
> loaded
> 2013-04-21 02:51:52,290 INFO plugin.PluginRepository - Plugins: looking
> in:
> /home/lailx/search_engine/nutch/plugins
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Plugin
> Auto-activation mode: [true]
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Registered Plugins:
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - the nutch
> core
> extension points (nutch-extensionpoints)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Regex URL
> Normalizer (urlnormalizer-regex)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - CyberNeko
> HTML
> Parser (lib-nekohtml)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - OPIC
> Scoring
> Plug-in (scoring-opic)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic URL
> Normalizer (urlnormalizer-basic)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Tika
> Parser Plug-in
> (parse-tika)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Html Parse
> Plug-in
> (parse-html)
> 2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic
> Indexing
> Filter (index-basic)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Anchor
> Indexing
> Filter (index-anchor)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - HTTP
> Framework
> (lib-http)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL
> Filter
> (urlfilter-regex)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL
> Filter
> Framework (lib-regex-filter)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository -
> Pass-through URL
> Normalizer (urlnormalizer-pass)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Http
> Protocol
> Plug-in (protocol-http)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Registered
> Extension-Points:
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL
> Normalizer (org.apache.nutch.net.URLNormalizer)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
> Protocol
> (org.apache.nutch.protocol.Protocol)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
> Segment Merge
> Filter (org.apache.nutch.segment.SegmentMergeFilter)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL
> Filter
> (org.apache.nutch.net.URLFilter)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
> Indexing
> Filter (org.apache.nutch.indexer.IndexingFilter)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - HTML Parse
> Filter
> (org.apache.nutch.parse.HtmlParseFilter)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
> Content
> Parser (org.apache.nutch.parse.Parser)
> 2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch
> Scoring
> (org.apache.nutch.scoring.ScoringFilter)
> 2013-04-21 02:51:52,866 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'inject', using default
> 2013-04-21 02:51:58,831 INFO crawl.Injector - Injector: total number of
> urls rejected by filters: 0
> 2013-04-21 02:51:58,831 INFO crawl.Injector - Injector: total number of
> urls injected after normalization and filtering: 1
> 2013-04-21 02:51:58,832 INFO crawl.Injector - Injector: Merging injected
> urls into crawl db.
> 2013-04-21 02:52:06,071 INFO crawl.Injector - Injector: finished at
> 2013-04-21 02:52:06, elapsed: 00:00:15
> 2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: starting at
> 2013-04-21 02:52:06
> 2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: filtering: true
> 2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: normalizing:
> true
> 2013-04-21 02:52:06,075 INFO crawl.Generator - Generator: topN: 5
> 2013-04-21 02:52:06,075 INFO crawl.Generator - Generator: jobtracker is
> 'local', generating exactly one partition.
> 2013-04-21 02:52:06,452 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:52:06,455 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:52:06,455 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:52:06,498 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 02:52:09,249 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:52:09,249 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:52:09,249 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:52:09,252 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'generate_host_count', using default
> 2013-04-21 02:52:13,206 INFO crawl.Generator - Generator: Partitioning
> selected urls for politeness.
> 2013-04-21 02:52:14,207 INFO crawl.Generator - Generator: segment:
> crawl/segments/20130421025214
> 2013-04-21 02:52:15,688 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 02:52:20,685 INFO crawl.Generator - Generator: finished at
> 2013-04-21 02:52:20, elapsed: 00:00:14
> 2013-04-21 02:52:20,686 WARN fetcher.Fetcher - Fetcher: Your
> 'http.agent.name' value should be listed first in 'http.robots.agents'
> property.
> 2013-04-21 02:52:20,687 INFO fetcher.Fetcher - Fetcher: starting at
> 2013-04-21 02:52:20
> 2013-04-21 02:52:20,687 INFO fetcher.Fetcher - Fetcher: segment:
> crawl/segments/20130421025214
> 2013-04-21 02:52:22,676 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,676 INFO fetcher.Fetcher - Fetcher: threads: 10
> 2013-04-21 02:52:22,677 INFO fetcher.Fetcher - Fetcher: time-out divisor:
> 2
> 2013-04-21 02:52:22,697 INFO fetcher.Fetcher - QueueFeeder finished: total
> 1 records + hit by time limit :0
> 2013-04-21 02:52:22,855 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,857 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,858 INFO fetcher.Fetcher - fetching
> http://www.163.com/
> 2013-04-21 02:52:22,859 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,859 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,859 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,860 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,861 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,861 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,862 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,862 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,868 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,868 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,871 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,876 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,877 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,889 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,890 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:52:22,893 INFO fetcher.Fetcher - Fetcher: throughput
> threshold: -1
> 2013-04-21 02:52:22,892 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,893 INFO fetcher.Fetcher - Fetcher: throughput
> threshold retries: 5
> 2013-04-21 02:52:22,894 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:52:22,908 INFO http.Http - http.proxy.host = null
> 2013-04-21 02:52:22,908 INFO http.Http - http.proxy.port = 8080
> 2013-04-21 02:52:22,908 INFO http.Http - http.timeout = 10000
> 2013-04-21 02:52:22,908 INFO http.Http - http.content.limit = 65536
> 2013-04-21 02:52:22,908 INFO http.Http - http.agent = myagent/Nutch-1.6
> 2013-04-21 02:52:22,908 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-04-21 02:52:22,908 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-04-21 02:52:23,896 INFO fetcher.Fetcher - -activeThreads=1,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 02:52:24,009 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=0
> 2013-04-21 02:52:24,897 INFO fetcher.Fetcher - -activeThreads=0,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 02:52:24,897 INFO fetcher.Fetcher - -activeThreads=0
> 2013-04-21 02:52:30,765 INFO fetcher.Fetcher - Fetcher: finished at
> 2013-04-21 02:52:30, elapsed: 00:00:10
> 2013-04-21 02:52:30,766 INFO parse.ParseSegment - ParseSegment: starting
> at
> 2013-04-21 02:52:30
> 2013-04-21 02:52:30,766 INFO parse.ParseSegment - ParseSegment: segment:
> crawl/segments/20130421025214
> 2013-04-21 02:52:39,879 INFO crawl.SignatureFactory - Using Signature
> impl:
> org.apache.nutch.crawl.MD5Signature
> 2013-04-21 02:52:39,895 INFO parse.ParseSegment - Parsed
> (21ms):http://www.163.com/
> 2013-04-21 02:52:40,394 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'outlink', using default
> 2013-04-21 02:52:43,796 INFO parse.ParseSegment - ParseSegment: finished
> at
> 2013-04-21 02:52:43, elapsed: 00:00:13
> 2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: starting at
> 2013-04-21 02:52:43
> 2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: db:
> crawl/crawldb
> 2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: segments:
> [crawl/segments/20130421025214]
> 2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: additions
> allowed: true
> 2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: URL
> normalizing: true
> 2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: URL
> filtering:
> true
> 2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
> false
> 2013-04-21 02:52:43,806 INFO crawl.CrawlDb - CrawlDb update: Merging
> segment data into db.
> 2013-04-21 02:52:44,000 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:52:47,060 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:52:50,034 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:52:52,930 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:52:52,930 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:52:52,930 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:52:56,900 INFO crawl.CrawlDb - CrawlDb update: finished at
> 2013-04-21 02:52:56, elapsed: 00:00:13
> 2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: starting at
> 2013-04-21 02:52:56
> 2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: filtering: true
> 2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: normalizing:
> true
> 2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: topN: 5
> 2013-04-21 02:52:56,905 INFO crawl.Generator - Generator: jobtracker is
> 'local', generating exactly one partition.
> 2013-04-21 02:52:57,079 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:52:57,079 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:52:57,080 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:52:57,081 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 02:52:59,983 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:52:59,983 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:52:59,983 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:52:59,984 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'generate_host_count', using default
> 2013-04-21 02:53:03,965 INFO crawl.Generator - Generator: Partitioning
> selected urls for politeness.
> 2013-04-21 02:53:04,966 INFO crawl.Generator - Generator: segment:
> crawl/segments/20130421025304
> 2013-04-21 02:53:05,164 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 02:53:12,037 INFO crawl.Generator - Generator: finished at
> 2013-04-21 02:53:12, elapsed: 00:00:15
> 2013-04-21 02:53:12,037 WARN fetcher.Fetcher - Fetcher: Your
> 'http.agent.name' value should be listed first in 'http.robots.agents'
> property.
> 2013-04-21 02:53:12,038 INFO fetcher.Fetcher - Fetcher: starting at
> 2013-04-21 02:53:12
> 2013-04-21 02:53:12,038 INFO fetcher.Fetcher - Fetcher: segment:
> crawl/segments/20130421025304
> 2013-04-21 02:53:12,384 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,385 INFO fetcher.Fetcher - Fetcher: threads: 10
> 2013-04-21 02:53:12,385 INFO fetcher.Fetcher - Fetcher: time-out divisor:
> 2
> 2013-04-21 02:53:12,391 INFO fetcher.Fetcher - QueueFeeder finished: total
> 5 records + hit by time limit :0
> 2013-04-21 02:53:12,397 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,398 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,398 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,399 INFO fetcher.Fetcher - fetching http://m.163.com/
> 2013-04-21 02:53:12,404 INFO http.Http - http.proxy.host = null
> 2013-04-21 02:53:12,404 INFO http.Http - http.proxy.port = 8080
> 2013-04-21 02:53:12,404 INFO http.Http - http.timeout = 10000
> 2013-04-21 02:53:12,404 INFO http.Http - http.content.limit = 65536
> 2013-04-21 02:53:12,404 INFO http.Http - http.agent = myagent/Nutch-1.6
> 2013-04-21 02:53:12,404 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-04-21 02:53:12,404 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-04-21 02:53:12,399 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,409 INFO fetcher.Fetcher - fetching
> http://3g.163.com/links/4145
> 2013-04-21 02:53:12,410 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,409 INFO fetcher.Fetcher - fetching
> http://music.163.com/
> 2013-04-21 02:53:12,411 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,412 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,413 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,419 INFO fetcher.Fetcher - fetching
> http://caipiao.163.com/mobile/client_cp.jsp
> 2013-04-21 02:53:12,422 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Fetcher: throughput
> threshold: -1
> 2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Fetcher: throughput
> threshold retries: 5
> 2013-04-21 02:53:12,651 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'fetcher', using default
> 2013-04-21 02:53:13,424 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=9, fetchQueues.totalSize=1
> 2013-04-21 02:53:13,424 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:13,425 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:13,425 INFO fetcher.Fetcher - inProgress = 1
> 2013-04-21 02:53:13,425 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:13,425 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:13,425 INFO fetcher.Fetcher - nextFetchTime =
> 1366483992387
> 2013-04-21 02:53:13,426 INFO fetcher.Fetcher - now =
> 1366483993426
> 2013-04-21 02:53:13,426 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:14,427 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=9, fetchQueues.totalSize=1
> 2013-04-21 02:53:14,427 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:14,427 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:14,427 INFO fetcher.Fetcher - inProgress = 1
> 2013-04-21 02:53:14,427 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:14,428 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:14,428 INFO fetcher.Fetcher - nextFetchTime =
> 1366483992387
> 2013-04-21 02:53:14,428 INFO fetcher.Fetcher - now =
> 1366483994428
> 2013-04-21 02:53:14,428 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:15,217 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'fetcher', using default
> 2013-04-21 02:53:15,428 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 02:53:15,429 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:15,429 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:15,429 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 02:53:15,429 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:15,430 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:15,430 INFO fetcher.Fetcher - nextFetchTime =
> 1366484000215
> 2013-04-21 02:53:15,430 INFO fetcher.Fetcher - now =
> 1366483995430
> 2013-04-21 02:53:15,430 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:16,431 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 02:53:16,431 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:16,431 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:16,431 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 02:53:16,432 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:16,432 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:16,432 INFO fetcher.Fetcher - nextFetchTime =
> 1366484000215
> 2013-04-21 02:53:16,432 INFO fetcher.Fetcher - now =
> 1366483996432
> 2013-04-21 02:53:16,432 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:17,433 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 02:53:17,433 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:17,433 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - nextFetchTime =
> 1366484000215
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - now =
> 1366483997434
> 2013-04-21 02:53:17,434 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:18,435 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 02:53:18,435 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:18,435 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - nextFetchTime =
> 1366484000215
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - now =
> 1366483998436
> 2013-04-21 02:53:18,436 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:19,437 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 02:53:19,437 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 02:53:19,437 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - nextFetchTime =
> 1366484000215
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - now =
> 1366483999438
> 2013-04-21 02:53:19,438 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 02:53:20,221 INFO fetcher.Fetcher - fetching
> http://m.163.com/newsapp/
> 2013-04-21 02:53:20,247 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=9
> 2013-04-21 02:53:20,330 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=8
> 2013-04-21 02:53:20,426 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=6
> 2013-04-21 02:53:20,426 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 02:53:20,427 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 02:53:20,428 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=4
> 2013-04-21 02:53:20,428 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=3
> 2013-04-21 02:53:20,429 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=2
> 2013-04-21 02:53:20,439 INFO fetcher.Fetcher - -activeThreads=2,
> spinWaiting=2, fetchQueues.totalSize=0
> 2013-04-21 02:53:20,548 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 02:53:20,580 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=0
> 2013-04-21 02:53:21,440 INFO fetcher.Fetcher - -activeThreads=0,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 02:53:21,440 INFO fetcher.Fetcher - -activeThreads=0
> 2013-04-21 02:53:28,097 INFO fetcher.Fetcher - Fetcher: finished at
> 2013-04-21 02:53:28, elapsed: 00:00:16
> 2013-04-21 02:53:28,098 INFO parse.ParseSegment - ParseSegment: starting
> at
> 2013-04-21 02:53:28
> 2013-04-21 02:53:28,098 INFO parse.ParseSegment - ParseSegment: segment:
> crawl/segments/20130421025304
> 2013-04-21 02:53:28,442 INFO crawl.SignatureFactory - Using Signature
> impl:
> org.apache.nutch.crawl.MD5Signature
> 2013-04-21 02:53:28,446 INFO parse.ParseSegment - Parsed
> (4ms):http://caipiao.163.com/mobile/client_cp.jsp
> 2013-04-21 02:53:28,544 INFO parse.ParseSegment - Parsed
> (3ms):http://m.163.com/newsapp/
> 2013-04-21 02:53:28,641 INFO parse.ParseSegment - Parsed
> (1ms):http://music.163.com/
> 2013-04-21 02:53:31,212 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'outlink', using default
> 2013-04-21 02:53:35,150 INFO parse.ParseSegment - ParseSegment: finished
> at
> 2013-04-21 02:53:35, elapsed: 00:00:07
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: starting at
> 2013-04-21 02:53:35
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: db:
> crawl/crawldb
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: segments:
> [crawl/segments/20130421025304]
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: additions
> allowed: true
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: URL
> normalizing: true
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: URL
> filtering:
> true
> 2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
> false
> 2013-04-21 02:53:35,154 INFO crawl.CrawlDb - CrawlDb update: Merging
> segment data into db.
> 2013-04-21 02:53:35,321 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:53:38,322 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:53:41,348 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 02:53:44,247 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 02:53:44,247 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 02:53:44,247 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 02:53:48,215 INFO crawl.CrawlDb - CrawlDb update: finished at
> 2013-04-21 02:53:48, elapsed: 00:00:13
> 2013-04-21 02:53:48,220 INFO crawl.LinkDb - LinkDb: starting at 2013-04-21
> 02:53:48
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: URL normalize: true
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: URL filter: true
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: internal links will be
> ignored.
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421025214
> 2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421025304
> 2013-04-21 02:53:48,436 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'linkdb', using default
> 2013-04-21 02:53:51,608 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'linkdb', using default
> 2013-04-21 02:53:58,277 INFO crawl.LinkDb - LinkDb: finished at 2013-04-21
> 02:53:58, elapsed: 00:00:10
> 2013-04-21 02:53:58,277 INFO crawl.Crawl - crawl finished: crawl
> 2013-04-21 16:03:44,720 WARN crawl.Crawl - solrUrl is not set, indexing
> will be skipped...
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - crawl started in: crawl
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - rootUrlDir = urls
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - threads = 10
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - depth = 3
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - solrUrl=null
> 2013-04-21 16:03:45,185 INFO crawl.Crawl - topN = 5
> 2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: starting at
> 2013-04-21 16:03:45
> 2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: crawlDb:
> crawl/crawldb
> 2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: urlDir: urls
> 2013-04-21 16:03:45,370 INFO crawl.Injector - Injector: Converting
> injected
> urls to crawl db entries.
> 2013-04-21 16:03:45,551 WARN util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2013-04-21 16:03:45,644 WARN snappy.LoadSnappy - Snappy native library not
> loaded
> 2013-04-21 16:03:46,453 INFO plugin.PluginRepository - Plugins: looking
> in:
> /home/lailx/search_engine/nutch/plugins
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Plugin
> Auto-activation mode: [true]
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Registered Plugins:
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - the nutch
> core
> extension points (nutch-extensionpoints)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Regex URL
> Normalizer (urlnormalizer-regex)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - CyberNeko
> HTML
> Parser (lib-nekohtml)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - OPIC
> Scoring
> Plug-in (scoring-opic)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Basic URL
> Normalizer (urlnormalizer-basic)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Tika
> Parser Plug-in
> (parse-tika)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Html Parse
> Plug-in
> (parse-html)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Basic
> Indexing
> Filter (index-basic)
> 2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Anchor
> Indexing
> Filter (index-anchor)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - HTTP
> Framework
> (lib-http)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Regex URL
> Filter
> (urlfilter-regex)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Regex URL
> Filter
> Framework (lib-regex-filter)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository -
> Pass-through URL
> Normalizer (urlnormalizer-pass)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Http
> Protocol
> Plug-in (protocol-http)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Registered
> Extension-Points:
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch URL
> Normalizer (org.apache.nutch.net.URLNormalizer)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch
> Protocol
> (org.apache.nutch.protocol.Protocol)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch
> Segment Merge
> Filter (org.apache.nutch.segment.SegmentMergeFilter)
> 2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch URL
> Filter
> (org.apache.nutch.net.URLFilter)
> 2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch
> Indexing
> Filter (org.apache.nutch.indexer.IndexingFilter)
> 2013-04-21 16:03:47,217 INFO plugin.PluginRepository - HTML Parse
> Filter
> (org.apache.nutch.parse.HtmlParseFilter)
> 2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch
> Content
> Parser (org.apache.nutch.parse.Parser)
> 2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch
> Scoring
> (org.apache.nutch.scoring.ScoringFilter)
> 2013-04-21 16:03:47,361 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'inject', using default
> 2013-04-21 16:03:53,030 INFO crawl.Injector - Injector: total number of
> urls rejected by filters: 0
> 2013-04-21 16:03:53,030 INFO crawl.Injector - Injector: total number of
> urls injected after normalization and filtering: 1
> 2013-04-21 16:03:53,031 INFO crawl.Injector - Injector: Merging injected
> urls into crawl db.
> 2013-04-21 16:04:00,199 INFO crawl.Injector - Injector: finished at
> 2013-04-21 16:04:00, elapsed: 00:00:14
> 2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: starting at
> 2013-04-21 16:04:00
> 2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: filtering: true
> 2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: normalizing:
> true
> 2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: topN: 5
> 2013-04-21 16:04:00,205 INFO crawl.Generator - Generator: jobtracker is
> 'local', generating exactly one partition.
> 2013-04-21 16:04:00,595 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:04:00,596 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:04:00,596 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:04:00,618 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:04:03,375 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:04:03,375 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:04:03,375 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:04:03,378 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'generate_host_count', using default
> 2013-04-21 16:04:07,330 INFO crawl.Generator - Generator: Partitioning
> selected urls for politeness.
> 2013-04-21 16:04:08,331 INFO crawl.Generator - Generator: segment:
> crawl/segments/20130421160408
> 2013-04-21 16:04:09,427 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:04:15,430 INFO crawl.Generator - Generator: finished at
> 2013-04-21 16:04:15, elapsed: 00:00:15
> 2013-04-21 16:04:15,431 WARN fetcher.Fetcher - Fetcher: Your
> 'http.agent.name' value should be listed first in 'http.robots.agents'
> property.
> 2013-04-21 16:04:15,431 INFO fetcher.Fetcher - Fetcher: starting at
> 2013-04-21 16:04:15
> 2013-04-21 16:04:15,431 INFO fetcher.Fetcher - Fetcher: segment:
> crawl/segments/20130421160408
> 2013-04-21 16:04:16,356 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,357 INFO fetcher.Fetcher - Fetcher: threads: 10
> 2013-04-21 16:04:16,357 INFO fetcher.Fetcher - Fetcher: time-out divisor:
> 2
> 2013-04-21 16:04:16,374 INFO fetcher.Fetcher - QueueFeeder finished: total
> 1 records + hit by time limit :0
> 2013-04-21 16:04:16,672 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,685 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,691 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,696 INFO fetcher.Fetcher - fetching
> http://www.163.com/
> 2013-04-21 16:04:16,698 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,698 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,702 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,712 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,715 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,716 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,716 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,722 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,727 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,727 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,735 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,743 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,751 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:04:16,752 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,756 INFO fetcher.Fetcher - Fetcher: throughput
> threshold: -1
> 2013-04-21 16:04:16,757 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,757 INFO fetcher.Fetcher - Fetcher: throughput
> threshold retries: 5
> 2013-04-21 16:04:16,756 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:04:16,891 INFO http.Http - http.proxy.host = null
> 2013-04-21 16:04:16,892 INFO http.Http - http.proxy.port = 8080
> 2013-04-21 16:04:16,892 INFO http.Http - http.timeout = 10000
> 2013-04-21 16:04:16,892 INFO http.Http - http.content.limit = 65536
> 2013-04-21 16:04:16,892 INFO http.Http - http.agent = myagent/Nutch-1.6
> 2013-04-21 16:04:16,892 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-04-21 16:04:16,892 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-04-21 16:04:17,767 INFO fetcher.Fetcher - -activeThreads=1,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 16:04:18,819 INFO fetcher.Fetcher - -activeThreads=1,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 16:04:19,066 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=0
> 2013-04-21 16:04:19,820 INFO fetcher.Fetcher - -activeThreads=0,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 16:04:19,820 INFO fetcher.Fetcher - -activeThreads=0
> 2013-04-21 16:04:25,826 INFO fetcher.Fetcher - Fetcher: finished at
> 2013-04-21 16:04:25, elapsed: 00:00:10
> 2013-04-21 16:04:25,827 INFO parse.ParseSegment - ParseSegment: starting
> at
> 2013-04-21 16:04:25
> 2013-04-21 16:04:25,827 INFO parse.ParseSegment - ParseSegment: segment:
> crawl/segments/20130421160408
> 2013-04-21 16:04:26,890 INFO crawl.SignatureFactory - Using Signature
> impl:
> org.apache.nutch.crawl.MD5Signature
> 2013-04-21 16:04:26,905 INFO parse.ParseSegment - Parsed
> (20ms):http://www.163.com/
> 2013-04-21 16:04:28,986 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'outlink', using default
> 2013-04-21 16:04:32,933 INFO parse.ParseSegment - ParseSegment: finished
> at
> 2013-04-21 16:04:32, elapsed: 00:00:07
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: starting at
> 2013-04-21 16:04:32
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: db:
> crawl/crawldb
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: segments:
> [crawl/segments/20130421160408]
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: additions
> allowed: true
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: URL
> normalizing: true
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: URL
> filtering:
> true
> 2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
> false
> 2013-04-21 16:04:32,937 INFO crawl.CrawlDb - CrawlDb update: Merging
> segment data into db.
> 2013-04-21 16:04:33,216 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:04:36,191 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:04:39,197 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:04:42,102 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:04:42,103 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:04:42,103 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:04:46,072 INFO crawl.CrawlDb - CrawlDb update: finished at
> 2013-04-21 16:04:46, elapsed: 00:00:13
> 2013-04-21 16:04:46,077 INFO crawl.Generator - Generator: starting at
> 2013-04-21 16:04:46
> 2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: filtering: true
> 2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: normalizing:
> true
> 2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: topN: 5
> 2013-04-21 16:04:46,079 INFO crawl.Generator - Generator: jobtracker is
> 'local', generating exactly one partition.
> 2013-04-21 16:04:48,690 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:04:48,690 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:04:48,690 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:04:48,691 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:04:49,172 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:04:49,172 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:04:49,172 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:04:49,174 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'generate_host_count', using default
> 2013-04-21 16:04:52,685 INFO crawl.Generator - Generator: Partitioning
> selected urls for politeness.
> 2013-04-21 16:04:53,686 INFO crawl.Generator - Generator: segment:
> crawl/segments/20130421160453
> 2013-04-21 16:04:54,225 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:05:00,805 INFO crawl.Generator - Generator: finished at
> 2013-04-21 16:05:00, elapsed: 00:00:14
> 2013-04-21 16:05:00,805 WARN fetcher.Fetcher - Fetcher: Your
> 'http.agent.name' value should be listed first in 'http.robots.agents'
> property.
> 2013-04-21 16:05:00,805 INFO fetcher.Fetcher - Fetcher: starting at
> 2013-04-21 16:05:00
> 2013-04-21 16:05:00,805 INFO fetcher.Fetcher - Fetcher: segment:
> crawl/segments/20130421160453
> 2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Fetcher: threads: 10
> 2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Fetcher: time-out divisor:
> 2
> 2013-04-21 16:05:01,006 INFO fetcher.Fetcher - QueueFeeder finished: total
> 5 records + hit by time limit :0
> 2013-04-21 16:05:01,021 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,022 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,023 INFO fetcher.Fetcher - fetching http://m.163.com/
> 2013-04-21 16:05:01,023 INFO http.Http - http.proxy.host = null
> 2013-04-21 16:05:01,030 INFO http.Http - http.proxy.port = 8080
> 2013-04-21 16:05:01,030 INFO http.Http - http.timeout = 10000
> 2013-04-21 16:05:01,030 INFO http.Http - http.content.limit = 65536
> 2013-04-21 16:05:01,030 INFO http.Http - http.agent = myagent/Nutch-1.6
> 2013-04-21 16:05:01,030 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-04-21 16:05:01,030 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-04-21 16:05:01,024 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,055 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,056 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,057 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,063 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,064 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,064 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,071 INFO fetcher.Fetcher - fetching
> http://music.163.com/
> 2013-04-21 16:05:01,069 INFO fetcher.Fetcher - fetching
> http://caipiao.163.com/mobile/client_cp.jsp
> 2013-04-21 16:05:01,068 INFO fetcher.Fetcher - fetching
> http://3g.163.com/links/4145
> 2013-04-21 16:05:01,087 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:01,088 INFO fetcher.Fetcher - Fetcher: throughput
> threshold: -1
> 2013-04-21 16:05:01,089 INFO fetcher.Fetcher - Fetcher: throughput
> threshold retries: 5
> 2013-04-21 16:05:01,265 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'fetcher', using default
> 2013-04-21 16:05:01,409 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'fetcher', using default
> 2013-04-21 16:05:02,090 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 16:05:02,090 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - nextFetchTime =
> 1366531506253
> 2013-04-21 16:05:02,091 INFO fetcher.Fetcher - now =
> 1366531502091
> 2013-04-21 16:05:02,092 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 16:05:03,092 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - nextFetchTime =
> 1366531506253
> 2013-04-21 16:05:03,093 INFO fetcher.Fetcher - now =
> 1366531503093
> 2013-04-21 16:05:03,094 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 16:05:04,094 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 16:05:04,095 INFO fetcher.Fetcher - nextFetchTime =
> 1366531506253
> 2013-04-21 16:05:04,095 INFO fetcher.Fetcher - now =
> 1366531504095
> 2013-04-21 16:05:04,095 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 16:05:05,096 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 16:05:05,096 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 16:05:05,096 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 16:05:05,096 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 16:05:05,097 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 16:05:05,097 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 16:05:05,097 INFO fetcher.Fetcher - nextFetchTime =
> 1366531506253
> 2013-04-21 16:05:05,097 INFO fetcher.Fetcher - now =
> 1366531505097
> 2013-04-21 16:05:05,097 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 16:05:06,098 INFO fetcher.Fetcher - -activeThreads=10,
> spinWaiting=10, fetchQueues.totalSize=1
> 2013-04-21 16:05:06,098 INFO fetcher.Fetcher - * queue: http://m.163.com
> 2013-04-21 16:05:06,098 INFO fetcher.Fetcher - maxThreads = 1
> 2013-04-21 16:05:06,098 INFO fetcher.Fetcher - inProgress = 0
> 2013-04-21 16:05:06,098 INFO fetcher.Fetcher - crawlDelay = 5000
> 2013-04-21 16:05:06,099 INFO fetcher.Fetcher - minCrawlDelay = 0
> 2013-04-21 16:05:06,099 INFO fetcher.Fetcher - nextFetchTime =
> 1366531506253
> 2013-04-21 16:05:06,099 INFO fetcher.Fetcher - now =
> 1366531506099
> 2013-04-21 16:05:06,099 INFO fetcher.Fetcher - 0.
> http://m.163.com/newsapp/
> 2013-04-21 16:05:06,264 INFO fetcher.Fetcher - fetching
> http://m.163.com/newsapp/
> 2013-04-21 16:05:06,399 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=9
> 2013-04-21 16:05:06,434 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=8
> 2013-04-21 16:05:06,511 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=7
> 2013-04-21 16:05:06,573 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 16:05:06,573 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 16:05:06,574 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=4
> 2013-04-21 16:05:06,575 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=3
> 2013-04-21 16:05:06,590 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=2
> 2013-04-21 16:05:06,591 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:05:06,722 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=0
> 2013-04-21 16:05:07,100 INFO fetcher.Fetcher - -activeThreads=0,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 16:05:07,100 INFO fetcher.Fetcher - -activeThreads=0
> 2013-04-21 16:05:13,874 INFO fetcher.Fetcher - Fetcher: finished at
> 2013-04-21 16:05:13, elapsed: 00:00:13
> 2013-04-21 16:05:13,874 INFO parse.ParseSegment - ParseSegment: starting
> at
> 2013-04-21 16:05:13
> 2013-04-21 16:05:13,875 INFO parse.ParseSegment - ParseSegment: segment:
> crawl/segments/20130421160453
> 2013-04-21 16:05:14,289 INFO crawl.SignatureFactory - Using Signature
> impl:
> org.apache.nutch.crawl.MD5Signature
> 2013-04-21 16:05:14,293 INFO parse.ParseSegment - Parsed
> (4ms):http://caipiao.163.com/mobile/client_cp.jsp
> 2013-04-21 16:05:14,421 INFO parse.ParseSegment - Parsed
> (4ms):http://m.163.com/newsapp/
> 2013-04-21 16:05:14,479 INFO parse.ParseSegment - Parsed
> (1ms):http://music.163.com/
> 2013-04-21 16:05:16,960 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'outlink', using default
> 2013-04-21 16:05:20,928 INFO parse.ParseSegment - ParseSegment: finished
> at
> 2013-04-21 16:05:20, elapsed: 00:00:07
> 2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: starting at
> 2013-04-21 16:05:20
> 2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: db:
> crawl/crawldb
> 2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: segments:
> [crawl/segments/20130421160453]
> 2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: additions
> allowed: true
> 2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: URL
> normalizing: true
> 2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: URL
> filtering:
> true
> 2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
> false
> 2013-04-21 16:05:20,936 INFO crawl.CrawlDb - CrawlDb update: Merging
> segment data into db.
> 2013-04-21 16:05:21,108 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:05:24,107 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:05:27,120 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:05:30,025 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:05:30,025 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:05:30,025 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:05:34,015 INFO crawl.CrawlDb - CrawlDb update: finished at
> 2013-04-21 16:05:34, elapsed: 00:00:13
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: starting at
> 2013-04-21 16:05:34
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: Selecting
> best-scoring urls due for fetch.
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: filtering: true
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: normalizing:
> true
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: topN: 5
> 2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: jobtracker is
> 'local', generating exactly one partition.
> 2013-04-21 16:05:34,363 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:05:34,363 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:05:34,363 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:05:34,364 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:05:37,077 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:05:37,077 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:05:37,077 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:05:37,078 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'generate_host_count', using default
> 2013-04-21 16:05:41,064 INFO crawl.Generator - Generator: Partitioning
> selected urls for politeness.
> 2013-04-21 16:05:42,065 INFO crawl.Generator - Generator: segment:
> crawl/segments/20130421160542
> 2013-04-21 16:05:42,232 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'partition', using default
> 2013-04-21 16:05:49,119 INFO crawl.Generator - Generator: finished at
> 2013-04-21 16:05:49, elapsed: 00:00:15
> 2013-04-21 16:05:49,120 WARN fetcher.Fetcher - Fetcher: Your
> 'http.agent.name' value should be listed first in 'http.robots.agents'
> property.
> 2013-04-21 16:05:49,120 INFO fetcher.Fetcher - Fetcher: starting at
> 2013-04-21 16:05:49
> 2013-04-21 16:05:49,120 INFO fetcher.Fetcher - Fetcher: segment:
> crawl/segments/20130421160542
> 2013-04-21 16:05:49,303 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,304 INFO fetcher.Fetcher - Fetcher: threads: 10
> 2013-04-21 16:05:49,304 INFO fetcher.Fetcher - Fetcher: time-out divisor:
> 2
> 2013-04-21 16:05:49,312 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,314 INFO fetcher.Fetcher - QueueFeeder finished: total
> 5 records + hit by time limit :0
> 2013-04-21 16:05:49,320 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,320 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,321 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,325 INFO fetcher.Fetcher - fetching
> http://help.163.com/
> 2013-04-21 16:05:49,326 INFO http.Http - http.proxy.host = null
> 2013-04-21 16:05:49,326 INFO http.Http - http.proxy.port = 8080
> 2013-04-21 16:05:49,326 INFO http.Http - http.timeout = 10000
> 2013-04-21 16:05:49,326 INFO http.Http - http.content.limit = 65536
> 2013-04-21 16:05:49,326 INFO http.Http - http.agent = myagent/Nutch-1.6
> 2013-04-21 16:05:49,326 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2013-04-21 16:05:49,326 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2013-04-21 16:05:49,331 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,332 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,332 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,333 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,334 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,335 INFO fetcher.Fetcher - Using queue mode : byHost
> 2013-04-21 16:05:49,339 INFO fetcher.Fetcher - fetching
> http://yuedu.163.com/
> 2013-04-21 16:05:49,340 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 16:05:49,340 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=6
> 2013-04-21 16:05:49,339 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=7
> 2013-04-21 16:05:49,341 INFO fetcher.Fetcher - Fetcher: throughput
> threshold: -1
> 2013-04-21 16:05:49,341 INFO fetcher.Fetcher - Fetcher: throughput
> threshold retries: 5
> 2013-04-21 16:05:49,341 INFO fetcher.Fetcher - fetching
> http://open.163.com/
> 2013-04-21 16:05:49,341 INFO fetcher.Fetcher - fetching
> http://reg.163.com/
> 2013-04-21 16:05:49,342 INFO fetcher.Fetcher - fetching
> http://sitemap.163.com/
> 2013-04-21 16:05:49,343 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 16:05:49,342 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=5
> 2013-04-21 16:05:49,407 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=4
> 2013-04-21 16:05:49,533 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=3
> 2013-04-21 16:05:49,568 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=2
> 2013-04-21 16:05:49,737 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=1
> 2013-04-21 16:05:49,943 INFO fetcher.Fetcher - -finishing thread
> FetcherThread, activeThreads=0
> 2013-04-21 16:05:50,341 INFO fetcher.Fetcher - -activeThreads=0,
> spinWaiting=0, fetchQueues.totalSize=0
> 2013-04-21 16:05:50,342 INFO fetcher.Fetcher - -activeThreads=0
> 2013-04-21 16:05:56,181 INFO fetcher.Fetcher - Fetcher: finished at
> 2013-04-21 16:05:56, elapsed: 00:00:07
> 2013-04-21 16:05:56,181 INFO parse.ParseSegment - ParseSegment: starting
> at
> 2013-04-21 16:05:56
> 2013-04-21 16:05:56,182 INFO parse.ParseSegment - ParseSegment: segment:
> crawl/segments/20130421160542
> 2013-04-21 16:05:56,489 INFO crawl.SignatureFactory - Using Signature
> impl:
> org.apache.nutch.crawl.MD5Signature
> 2013-04-21 16:05:56,497 INFO parse.ParseSegment - Parsed
> (8ms):http://help.163.com/
> 2013-04-21 16:05:56,627 INFO parse.ParseSegment - Parsed
> (9ms):http://open.163.com/
> 2013-04-21 16:05:56,697 INFO parse.ParseSegment - Parsed
> (3ms):http://reg.163.com/
> 2013-04-21 16:05:56,827 INFO parse.ParseSegment - Parsed
> (6ms):http://sitemap.163.com/
> 2013-04-21 16:05:56,952 INFO parse.ParseSegment - Parsed
> (4ms):http://yuedu.163.com/
> 2013-04-21 16:05:59,272 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'outlink', using default
> 2013-04-21 16:06:03,234 INFO parse.ParseSegment - ParseSegment: finished
> at
> 2013-04-21 16:06:03, elapsed: 00:00:07
> 2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: starting at
> 2013-04-21 16:06:03
> 2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: db:
> crawl/crawldb
> 2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: segments:
> [crawl/segments/20130421160542]
> 2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: additions
> allowed: true
> 2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: URL
> normalizing: true
> 2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: URL
> filtering:
> true
> 2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
> false
> 2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: Merging
> segment data into db.
> 2013-04-21 16:06:03,419 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:06:06,450 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:06:09,408 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'crawldb', using default
> 2013-04-21 16:06:12,341 INFO crawl.FetchScheduleFactory - Using
> FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
> 2013-04-21 16:06:12,341 INFO crawl.AbstractFetchSchedule -
> defaultInterval=2592000
> 2013-04-21 16:06:12,341 INFO crawl.AbstractFetchSchedule -
> maxInterval=7776000
> 2013-04-21 16:06:16,324 INFO crawl.CrawlDb - CrawlDb update: finished at
> 2013-04-21 16:06:16, elapsed: 00:00:13
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: starting at 2013-04-21
> 16:06:16
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: URL normalize: true
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: URL filter: true
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: internal links will be
> ignored.
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421160453
> 2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421160408
> 2013-04-21 16:06:16,331 INFO crawl.LinkDb - LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421160542
> 2013-04-21 16:06:16,483 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'linkdb', using default
> 2013-04-21 16:06:19,464 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'linkdb', using default
> 2013-04-21 16:06:22,498 INFO regex.RegexURLNormalizer - can't find rules
> for scope 'linkdb', using default
> 2013-04-21 16:06:29,386 INFO crawl.LinkDb - LinkDb: finished at 2013-04-21
> 16:06:29, elapsed: 00:00:13
> 2013-04-21 16:06:29,387 INFO crawl.Crawl - crawl finished: crawl
> 2013-04-21 16:06:57,720 INFO solr.SolrIndexer - SolrIndexer: starting at
> 2013-04-21 16:06:57
> 2013-04-21 16:06:57,892 INFO solr.SolrIndexer - SolrIndexer: deleting gone
> documents: false
> 2013-04-21 16:06:57,893 INFO solr.SolrIndexer - SolrIndexer: URL
> filtering:
> false
> 2013-04-21 16:06:57,893 INFO solr.SolrIndexer - SolrIndexer: URL
> normalizing: false
> 2013-04-21 16:06:57,915 INFO indexer.IndexerMapReduce - IndexerMapReduce:
> crawldb: crawl/crawldb
> 2013-04-21 16:06:57,916 INFO indexer.IndexerMapReduce - IndexerMapReduce:
> linkdb: crawl/linkdb
> 2013-04-21 16:06:57,916 INFO indexer.IndexerMapReduce - IndexerMapReduces:
> adding segment: crawl/segments/20130421160408
> 2013-04-21 16:06:58,661 INFO indexer.IndexerMapReduce - IndexerMapReduces:
> adding segment: crawl/segments/20130421160453
> 2013-04-21 16:06:58,663 INFO indexer.IndexerMapReduce - IndexerMapReduces:
> adding segment: crawl/segments/20130421160542
> 2013-04-21 16:06:58,940 WARN util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2013-04-21 16:07:00,267 INFO plugin.PluginRepository - Plugins: looking
> in:
> /home/lailx/search_engine/nutch/plugins
> 2013-04-21 16:07:00,604 INFO plugin.PluginRepository - Plugin
> Auto-activation mode: [true]
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Registered Plugins:
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - the nutch
> core
> extension points (nutch-extensionpoints)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL
> Normalizer (urlnormalizer-regex)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - CyberNeko
> HTML
> Parser (lib-nekohtml)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - OPIC
> Scoring
> Plug-in (scoring-opic)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Basic URL
> Normalizer (urlnormalizer-basic)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Tika
> Parser Plug-in
> (parse-tika)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Html Parse
> Plug-in
> (parse-html)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Basic
> Indexing
> Filter (index-basic)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Anchor
> Indexing
> Filter (index-anchor)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - HTTP
> Framework
> (lib-http)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL
> Filter
> (urlfilter-regex)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL
> Filter
> Framework (lib-regex-filter)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository -
> Pass-through URL
> Normalizer (urlnormalizer-pass)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Http
> Protocol
> Plug-in (protocol-http)
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Registered
> Extension-Points:
> 2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Nutch URL
> Normalizer (org.apache.nutch.net.URLNormalizer)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch
> Protocol
> (org.apache.nutch.protocol.Protocol)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch
> Segment Merge
> Filter (org.apache.nutch.segment.SegmentMergeFilter)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch URL
> Filter
> (org.apache.nutch.net.URLFilter)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch
> Indexing
> Filter (org.apache.nutch.indexer.IndexingFilter)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - HTML Parse
> Filter
> (org.apache.nutch.parse.HtmlParseFilter)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch
> Content
> Parser (org.apache.nutch.parse.Parser)
> 2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch
> Scoring
> (org.apache.nutch.scoring.ScoringFilter)
> 2013-04-21 16:07:00,616 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:00,619 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:00,620 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:02,976 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:02,976 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:02,976 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:05,985 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:05,985 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:05,985 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:09,014 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:09,014 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:09,014 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:12,065 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:12,065 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:12,065 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:15,006 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:15,010 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:15,010 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:18,034 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:18,034 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:18,034 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:21,031 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:21,031 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:21,031 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:24,043 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:24,043 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:24,043 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:27,058 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:27,058 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:27,058 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:30,189 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:30,189 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:30,189 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:33,063 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:33,064 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:33,064 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:36,099 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:36,099 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:36,099 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:39,087 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:39,088 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:39,088 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:42,029 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2013-04-21 16:07:42,029 INFO anchor.AnchorIndexingFilter - Anchor
> deduplication is: off
> 2013-04-21 16:07:42,029 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: content
> dest:
> content
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: title dest:
> title
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: host dest:
> host
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: segment
> dest:
> segment
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: boost dest:
> boost
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: digest dest:
> digest
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: tstamp dest:
> tstamp
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: url dest: id
> 2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: url dest:
> url
> 2013-04-21 16:07:42,308 INFO solr.SolrWriter - Indexing 9 documents
> 2013-04-21 16:07:43,367 WARN mapred.LocalJobRunner - job_local_0001
> *org.apache.solr.common.SolrException: Bad Request
>
> Bad Request
>
> request: http://127.0.0.1:8080/solr/update?wt=javabin&version=2
> at
>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
> at
>
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
> at
>
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at
> org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:142)
> at
>
> org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
> at
>
> org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.close(ReduceTask.java:466)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
> 2013-04-21 16:07:43,699 ERROR solr.SolrIndexer - java.io.IOException: Job
> failed! *
> Is the 'Bad Request' caused the exception?
> Waiting your reply.
> Thanks in advance.
>
> note: I used solr-4.2.0 this time, still not work.
>
> Br,
> Mick
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620p4057688.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
Kiran Chitturi
<http://www.linkedin.com/in/kiranchitturi>
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by micklai <la...@gmail.com>.
HI Kiran,
Thank you for you support, but since practiced by following NutchTutorial,
it still failed, please refer to the hadoop.log below....
2013-04-21 02:51:50,025 WARN crawl.Crawl - solrUrl is not set, indexing
will be skipped...
2013-04-21 02:51:50,679 INFO crawl.Crawl - crawl started in: crawl
2013-04-21 02:51:50,679 INFO crawl.Crawl - rootUrlDir = urls
2013-04-21 02:51:50,680 INFO crawl.Crawl - threads = 10
2013-04-21 02:51:50,680 INFO crawl.Crawl - depth = 2
2013-04-21 02:51:50,680 INFO crawl.Crawl - solrUrl=null
2013-04-21 02:51:50,680 INFO crawl.Crawl - topN = 5
2013-04-21 02:51:50,765 INFO crawl.Injector - Injector: starting at
2013-04-21 02:51:50
2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: crawlDb:
crawl/crawldb
2013-04-21 02:51:50,766 INFO crawl.Injector - Injector: urlDir: urls
2013-04-21 02:51:50,848 INFO crawl.Injector - Injector: Converting injected
urls to crawl db entries.
2013-04-21 02:51:51,158 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2013-04-21 02:51:51,331 WARN snappy.LoadSnappy - Snappy native library not
loaded
2013-04-21 02:51:52,290 INFO plugin.PluginRepository - Plugins: looking in:
/home/lailx/search_engine/nutch/plugins
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Registered Plugins:
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - the nutch core
extension points (nutch-extensionpoints)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Regex URL
Normalizer (urlnormalizer-regex)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - CyberNeko HTML
Parser (lib-nekohtml)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic URL
Normalizer (urlnormalizer-basic)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Tika Parser Plug-in
(parse-tika)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Html Parse Plug-in
(parse-html)
2013-04-21 02:51:52,755 INFO plugin.PluginRepository - Basic Indexing
Filter (index-basic)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Anchor Indexing
Filter (index-anchor)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - HTTP Framework
(lib-http)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL Filter
(urlfilter-regex)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Regex URL Filter
Framework (lib-regex-filter)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Pass-through URL
Normalizer (urlnormalizer-pass)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Http Protocol
Plug-in (protocol-http)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Registered
Extension-Points:
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch Protocol
(org.apache.nutch.protocol.Protocol)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch Segment Merge
Filter (org.apache.nutch.segment.SegmentMergeFilter)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch URL Filter
(org.apache.nutch.net.URLFilter)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch Indexing
Filter (org.apache.nutch.indexer.IndexingFilter)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - HTML Parse Filter
(org.apache.nutch.parse.HtmlParseFilter)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch Content
Parser (org.apache.nutch.parse.Parser)
2013-04-21 02:51:52,756 INFO plugin.PluginRepository - Nutch Scoring
(org.apache.nutch.scoring.ScoringFilter)
2013-04-21 02:51:52,866 INFO regex.RegexURLNormalizer - can't find rules
for scope 'inject', using default
2013-04-21 02:51:58,831 INFO crawl.Injector - Injector: total number of
urls rejected by filters: 0
2013-04-21 02:51:58,831 INFO crawl.Injector - Injector: total number of
urls injected after normalization and filtering: 1
2013-04-21 02:51:58,832 INFO crawl.Injector - Injector: Merging injected
urls into crawl db.
2013-04-21 02:52:06,071 INFO crawl.Injector - Injector: finished at
2013-04-21 02:52:06, elapsed: 00:00:15
2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: starting at
2013-04-21 02:52:06
2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: filtering: true
2013-04-21 02:52:06,074 INFO crawl.Generator - Generator: normalizing: true
2013-04-21 02:52:06,075 INFO crawl.Generator - Generator: topN: 5
2013-04-21 02:52:06,075 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2013-04-21 02:52:06,452 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:52:06,455 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:52:06,455 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:52:06,498 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 02:52:09,249 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:52:09,249 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:52:09,249 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:52:09,252 INFO regex.RegexURLNormalizer - can't find rules
for scope 'generate_host_count', using default
2013-04-21 02:52:13,206 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2013-04-21 02:52:14,207 INFO crawl.Generator - Generator: segment:
crawl/segments/20130421025214
2013-04-21 02:52:15,688 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 02:52:20,685 INFO crawl.Generator - Generator: finished at
2013-04-21 02:52:20, elapsed: 00:00:14
2013-04-21 02:52:20,686 WARN fetcher.Fetcher - Fetcher: Your
'http.agent.name' value should be listed first in 'http.robots.agents'
property.
2013-04-21 02:52:20,687 INFO fetcher.Fetcher - Fetcher: starting at
2013-04-21 02:52:20
2013-04-21 02:52:20,687 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20130421025214
2013-04-21 02:52:22,676 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,676 INFO fetcher.Fetcher - Fetcher: threads: 10
2013-04-21 02:52:22,677 INFO fetcher.Fetcher - Fetcher: time-out divisor: 2
2013-04-21 02:52:22,697 INFO fetcher.Fetcher - QueueFeeder finished: total
1 records + hit by time limit :0
2013-04-21 02:52:22,855 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,857 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,858 INFO fetcher.Fetcher - fetching http://www.163.com/
2013-04-21 02:52:22,859 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,859 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,859 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,860 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,861 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,861 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,862 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,862 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,868 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,868 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,871 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,876 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,877 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,889 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,890 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:52:22,893 INFO fetcher.Fetcher - Fetcher: throughput
threshold: -1
2013-04-21 02:52:22,892 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,893 INFO fetcher.Fetcher - Fetcher: throughput
threshold retries: 5
2013-04-21 02:52:22,894 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:52:22,908 INFO http.Http - http.proxy.host = null
2013-04-21 02:52:22,908 INFO http.Http - http.proxy.port = 8080
2013-04-21 02:52:22,908 INFO http.Http - http.timeout = 10000
2013-04-21 02:52:22,908 INFO http.Http - http.content.limit = 65536
2013-04-21 02:52:22,908 INFO http.Http - http.agent = myagent/Nutch-1.6
2013-04-21 02:52:22,908 INFO http.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2013-04-21 02:52:22,908 INFO http.Http - http.accept =
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2013-04-21 02:52:23,896 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 02:52:24,009 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2013-04-21 02:52:24,897 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 02:52:24,897 INFO fetcher.Fetcher - -activeThreads=0
2013-04-21 02:52:30,765 INFO fetcher.Fetcher - Fetcher: finished at
2013-04-21 02:52:30, elapsed: 00:00:10
2013-04-21 02:52:30,766 INFO parse.ParseSegment - ParseSegment: starting at
2013-04-21 02:52:30
2013-04-21 02:52:30,766 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20130421025214
2013-04-21 02:52:39,879 INFO crawl.SignatureFactory - Using Signature impl:
org.apache.nutch.crawl.MD5Signature
2013-04-21 02:52:39,895 INFO parse.ParseSegment - Parsed
(21ms):http://www.163.com/
2013-04-21 02:52:40,394 INFO regex.RegexURLNormalizer - can't find rules
for scope 'outlink', using default
2013-04-21 02:52:43,796 INFO parse.ParseSegment - ParseSegment: finished at
2013-04-21 02:52:43, elapsed: 00:00:13
2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: starting at
2013-04-21 02:52:43
2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20130421025214]
2013-04-21 02:52:43,804 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2013-04-21 02:52:43,805 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
false
2013-04-21 02:52:43,806 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2013-04-21 02:52:44,000 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:52:47,060 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:52:50,034 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:52:52,930 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:52:52,930 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:52:52,930 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:52:56,900 INFO crawl.CrawlDb - CrawlDb update: finished at
2013-04-21 02:52:56, elapsed: 00:00:13
2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: starting at
2013-04-21 02:52:56
2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: filtering: true
2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: normalizing: true
2013-04-21 02:52:56,904 INFO crawl.Generator - Generator: topN: 5
2013-04-21 02:52:56,905 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2013-04-21 02:52:57,079 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:52:57,079 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:52:57,080 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:52:57,081 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 02:52:59,983 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:52:59,983 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:52:59,983 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:52:59,984 INFO regex.RegexURLNormalizer - can't find rules
for scope 'generate_host_count', using default
2013-04-21 02:53:03,965 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2013-04-21 02:53:04,966 INFO crawl.Generator - Generator: segment:
crawl/segments/20130421025304
2013-04-21 02:53:05,164 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 02:53:12,037 INFO crawl.Generator - Generator: finished at
2013-04-21 02:53:12, elapsed: 00:00:15
2013-04-21 02:53:12,037 WARN fetcher.Fetcher - Fetcher: Your
'http.agent.name' value should be listed first in 'http.robots.agents'
property.
2013-04-21 02:53:12,038 INFO fetcher.Fetcher - Fetcher: starting at
2013-04-21 02:53:12
2013-04-21 02:53:12,038 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20130421025304
2013-04-21 02:53:12,384 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,385 INFO fetcher.Fetcher - Fetcher: threads: 10
2013-04-21 02:53:12,385 INFO fetcher.Fetcher - Fetcher: time-out divisor: 2
2013-04-21 02:53:12,391 INFO fetcher.Fetcher - QueueFeeder finished: total
5 records + hit by time limit :0
2013-04-21 02:53:12,397 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,398 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,398 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,399 INFO fetcher.Fetcher - fetching http://m.163.com/
2013-04-21 02:53:12,404 INFO http.Http - http.proxy.host = null
2013-04-21 02:53:12,404 INFO http.Http - http.proxy.port = 8080
2013-04-21 02:53:12,404 INFO http.Http - http.timeout = 10000
2013-04-21 02:53:12,404 INFO http.Http - http.content.limit = 65536
2013-04-21 02:53:12,404 INFO http.Http - http.agent = myagent/Nutch-1.6
2013-04-21 02:53:12,404 INFO http.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2013-04-21 02:53:12,404 INFO http.Http - http.accept =
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2013-04-21 02:53:12,399 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,409 INFO fetcher.Fetcher - fetching
http://3g.163.com/links/4145
2013-04-21 02:53:12,410 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,409 INFO fetcher.Fetcher - fetching
http://music.163.com/
2013-04-21 02:53:12,411 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,412 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,413 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,419 INFO fetcher.Fetcher - fetching
http://caipiao.163.com/mobile/client_cp.jsp
2013-04-21 02:53:12,422 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Fetcher: throughput
threshold: -1
2013-04-21 02:53:12,423 INFO fetcher.Fetcher - Fetcher: throughput
threshold retries: 5
2013-04-21 02:53:12,651 INFO regex.RegexURLNormalizer - can't find rules
for scope 'fetcher', using default
2013-04-21 02:53:13,424 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=9, fetchQueues.totalSize=1
2013-04-21 02:53:13,424 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:13,425 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:13,425 INFO fetcher.Fetcher - inProgress = 1
2013-04-21 02:53:13,425 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:13,425 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:13,425 INFO fetcher.Fetcher - nextFetchTime =
1366483992387
2013-04-21 02:53:13,426 INFO fetcher.Fetcher - now =
1366483993426
2013-04-21 02:53:13,426 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:14,427 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=9, fetchQueues.totalSize=1
2013-04-21 02:53:14,427 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:14,427 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:14,427 INFO fetcher.Fetcher - inProgress = 1
2013-04-21 02:53:14,427 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:14,428 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:14,428 INFO fetcher.Fetcher - nextFetchTime =
1366483992387
2013-04-21 02:53:14,428 INFO fetcher.Fetcher - now =
1366483994428
2013-04-21 02:53:14,428 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:15,217 INFO regex.RegexURLNormalizer - can't find rules
for scope 'fetcher', using default
2013-04-21 02:53:15,428 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 02:53:15,429 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:15,429 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:15,429 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 02:53:15,429 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:15,430 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:15,430 INFO fetcher.Fetcher - nextFetchTime =
1366484000215
2013-04-21 02:53:15,430 INFO fetcher.Fetcher - now =
1366483995430
2013-04-21 02:53:15,430 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:16,431 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 02:53:16,431 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:16,431 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:16,431 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 02:53:16,432 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:16,432 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:16,432 INFO fetcher.Fetcher - nextFetchTime =
1366484000215
2013-04-21 02:53:16,432 INFO fetcher.Fetcher - now =
1366483996432
2013-04-21 02:53:16,432 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:17,433 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 02:53:17,433 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:17,433 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - nextFetchTime =
1366484000215
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - now =
1366483997434
2013-04-21 02:53:17,434 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:18,435 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 02:53:18,435 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:18,435 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - nextFetchTime =
1366484000215
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - now =
1366483998436
2013-04-21 02:53:18,436 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:19,437 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 02:53:19,437 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 02:53:19,437 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - nextFetchTime =
1366484000215
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - now =
1366483999438
2013-04-21 02:53:19,438 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 02:53:20,221 INFO fetcher.Fetcher - fetching
http://m.163.com/newsapp/
2013-04-21 02:53:20,247 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=9
2013-04-21 02:53:20,330 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=8
2013-04-21 02:53:20,426 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=6
2013-04-21 02:53:20,426 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 02:53:20,427 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 02:53:20,428 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=4
2013-04-21 02:53:20,428 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=3
2013-04-21 02:53:20,429 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2013-04-21 02:53:20,439 INFO fetcher.Fetcher - -activeThreads=2,
spinWaiting=2, fetchQueues.totalSize=0
2013-04-21 02:53:20,548 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 02:53:20,580 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2013-04-21 02:53:21,440 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 02:53:21,440 INFO fetcher.Fetcher - -activeThreads=0
2013-04-21 02:53:28,097 INFO fetcher.Fetcher - Fetcher: finished at
2013-04-21 02:53:28, elapsed: 00:00:16
2013-04-21 02:53:28,098 INFO parse.ParseSegment - ParseSegment: starting at
2013-04-21 02:53:28
2013-04-21 02:53:28,098 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20130421025304
2013-04-21 02:53:28,442 INFO crawl.SignatureFactory - Using Signature impl:
org.apache.nutch.crawl.MD5Signature
2013-04-21 02:53:28,446 INFO parse.ParseSegment - Parsed
(4ms):http://caipiao.163.com/mobile/client_cp.jsp
2013-04-21 02:53:28,544 INFO parse.ParseSegment - Parsed
(3ms):http://m.163.com/newsapp/
2013-04-21 02:53:28,641 INFO parse.ParseSegment - Parsed
(1ms):http://music.163.com/
2013-04-21 02:53:31,212 INFO regex.RegexURLNormalizer - can't find rules
for scope 'outlink', using default
2013-04-21 02:53:35,150 INFO parse.ParseSegment - ParseSegment: finished at
2013-04-21 02:53:35, elapsed: 00:00:07
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: starting at
2013-04-21 02:53:35
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20130421025304]
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2013-04-21 02:53:35,153 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
false
2013-04-21 02:53:35,154 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2013-04-21 02:53:35,321 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:53:38,322 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:53:41,348 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 02:53:44,247 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 02:53:44,247 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 02:53:44,247 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 02:53:48,215 INFO crawl.CrawlDb - CrawlDb update: finished at
2013-04-21 02:53:48, elapsed: 00:00:13
2013-04-21 02:53:48,220 INFO crawl.LinkDb - LinkDb: starting at 2013-04-21
02:53:48
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: URL normalize: true
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: URL filter: true
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: internal links will be
ignored.
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421025214
2013-04-21 02:53:48,221 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421025304
2013-04-21 02:53:48,436 INFO regex.RegexURLNormalizer - can't find rules
for scope 'linkdb', using default
2013-04-21 02:53:51,608 INFO regex.RegexURLNormalizer - can't find rules
for scope 'linkdb', using default
2013-04-21 02:53:58,277 INFO crawl.LinkDb - LinkDb: finished at 2013-04-21
02:53:58, elapsed: 00:00:10
2013-04-21 02:53:58,277 INFO crawl.Crawl - crawl finished: crawl
2013-04-21 16:03:44,720 WARN crawl.Crawl - solrUrl is not set, indexing
will be skipped...
2013-04-21 16:03:45,185 INFO crawl.Crawl - crawl started in: crawl
2013-04-21 16:03:45,185 INFO crawl.Crawl - rootUrlDir = urls
2013-04-21 16:03:45,185 INFO crawl.Crawl - threads = 10
2013-04-21 16:03:45,185 INFO crawl.Crawl - depth = 3
2013-04-21 16:03:45,185 INFO crawl.Crawl - solrUrl=null
2013-04-21 16:03:45,185 INFO crawl.Crawl - topN = 5
2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: starting at
2013-04-21 16:03:45
2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: crawlDb:
crawl/crawldb
2013-04-21 16:03:45,269 INFO crawl.Injector - Injector: urlDir: urls
2013-04-21 16:03:45,370 INFO crawl.Injector - Injector: Converting injected
urls to crawl db entries.
2013-04-21 16:03:45,551 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2013-04-21 16:03:45,644 WARN snappy.LoadSnappy - Snappy native library not
loaded
2013-04-21 16:03:46,453 INFO plugin.PluginRepository - Plugins: looking in:
/home/lailx/search_engine/nutch/plugins
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Registered Plugins:
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - the nutch core
extension points (nutch-extensionpoints)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Regex URL
Normalizer (urlnormalizer-regex)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - CyberNeko HTML
Parser (lib-nekohtml)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Basic URL
Normalizer (urlnormalizer-basic)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Tika Parser Plug-in
(parse-tika)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Html Parse Plug-in
(parse-html)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Basic Indexing
Filter (index-basic)
2013-04-21 16:03:47,215 INFO plugin.PluginRepository - Anchor Indexing
Filter (index-anchor)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - HTTP Framework
(lib-http)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Regex URL Filter
(urlfilter-regex)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Regex URL Filter
Framework (lib-regex-filter)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Pass-through URL
Normalizer (urlnormalizer-pass)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Http Protocol
Plug-in (protocol-http)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Registered
Extension-Points:
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch Protocol
(org.apache.nutch.protocol.Protocol)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch Segment Merge
Filter (org.apache.nutch.segment.SegmentMergeFilter)
2013-04-21 16:03:47,216 INFO plugin.PluginRepository - Nutch URL Filter
(org.apache.nutch.net.URLFilter)
2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch Indexing
Filter (org.apache.nutch.indexer.IndexingFilter)
2013-04-21 16:03:47,217 INFO plugin.PluginRepository - HTML Parse Filter
(org.apache.nutch.parse.HtmlParseFilter)
2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch Content
Parser (org.apache.nutch.parse.Parser)
2013-04-21 16:03:47,217 INFO plugin.PluginRepository - Nutch Scoring
(org.apache.nutch.scoring.ScoringFilter)
2013-04-21 16:03:47,361 INFO regex.RegexURLNormalizer - can't find rules
for scope 'inject', using default
2013-04-21 16:03:53,030 INFO crawl.Injector - Injector: total number of
urls rejected by filters: 0
2013-04-21 16:03:53,030 INFO crawl.Injector - Injector: total number of
urls injected after normalization and filtering: 1
2013-04-21 16:03:53,031 INFO crawl.Injector - Injector: Merging injected
urls into crawl db.
2013-04-21 16:04:00,199 INFO crawl.Injector - Injector: finished at
2013-04-21 16:04:00, elapsed: 00:00:14
2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: starting at
2013-04-21 16:04:00
2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: filtering: true
2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: normalizing: true
2013-04-21 16:04:00,204 INFO crawl.Generator - Generator: topN: 5
2013-04-21 16:04:00,205 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2013-04-21 16:04:00,595 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:04:00,596 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:04:00,596 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:04:00,618 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:04:03,375 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:04:03,375 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:04:03,375 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:04:03,378 INFO regex.RegexURLNormalizer - can't find rules
for scope 'generate_host_count', using default
2013-04-21 16:04:07,330 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2013-04-21 16:04:08,331 INFO crawl.Generator - Generator: segment:
crawl/segments/20130421160408
2013-04-21 16:04:09,427 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:04:15,430 INFO crawl.Generator - Generator: finished at
2013-04-21 16:04:15, elapsed: 00:00:15
2013-04-21 16:04:15,431 WARN fetcher.Fetcher - Fetcher: Your
'http.agent.name' value should be listed first in 'http.robots.agents'
property.
2013-04-21 16:04:15,431 INFO fetcher.Fetcher - Fetcher: starting at
2013-04-21 16:04:15
2013-04-21 16:04:15,431 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20130421160408
2013-04-21 16:04:16,356 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,357 INFO fetcher.Fetcher - Fetcher: threads: 10
2013-04-21 16:04:16,357 INFO fetcher.Fetcher - Fetcher: time-out divisor: 2
2013-04-21 16:04:16,374 INFO fetcher.Fetcher - QueueFeeder finished: total
1 records + hit by time limit :0
2013-04-21 16:04:16,672 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,685 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,691 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,696 INFO fetcher.Fetcher - fetching http://www.163.com/
2013-04-21 16:04:16,698 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,698 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,702 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,712 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,715 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,716 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,716 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,722 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,727 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,727 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,735 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,743 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,751 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:04:16,752 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,756 INFO fetcher.Fetcher - Fetcher: throughput
threshold: -1
2013-04-21 16:04:16,757 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,757 INFO fetcher.Fetcher - Fetcher: throughput
threshold retries: 5
2013-04-21 16:04:16,756 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:04:16,891 INFO http.Http - http.proxy.host = null
2013-04-21 16:04:16,892 INFO http.Http - http.proxy.port = 8080
2013-04-21 16:04:16,892 INFO http.Http - http.timeout = 10000
2013-04-21 16:04:16,892 INFO http.Http - http.content.limit = 65536
2013-04-21 16:04:16,892 INFO http.Http - http.agent = myagent/Nutch-1.6
2013-04-21 16:04:16,892 INFO http.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2013-04-21 16:04:16,892 INFO http.Http - http.accept =
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2013-04-21 16:04:17,767 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 16:04:18,819 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 16:04:19,066 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2013-04-21 16:04:19,820 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 16:04:19,820 INFO fetcher.Fetcher - -activeThreads=0
2013-04-21 16:04:25,826 INFO fetcher.Fetcher - Fetcher: finished at
2013-04-21 16:04:25, elapsed: 00:00:10
2013-04-21 16:04:25,827 INFO parse.ParseSegment - ParseSegment: starting at
2013-04-21 16:04:25
2013-04-21 16:04:25,827 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20130421160408
2013-04-21 16:04:26,890 INFO crawl.SignatureFactory - Using Signature impl:
org.apache.nutch.crawl.MD5Signature
2013-04-21 16:04:26,905 INFO parse.ParseSegment - Parsed
(20ms):http://www.163.com/
2013-04-21 16:04:28,986 INFO regex.RegexURLNormalizer - can't find rules
for scope 'outlink', using default
2013-04-21 16:04:32,933 INFO parse.ParseSegment - ParseSegment: finished at
2013-04-21 16:04:32, elapsed: 00:00:07
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: starting at
2013-04-21 16:04:32
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20130421160408]
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2013-04-21 16:04:32,936 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
false
2013-04-21 16:04:32,937 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2013-04-21 16:04:33,216 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:04:36,191 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:04:39,197 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:04:42,102 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:04:42,103 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:04:42,103 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:04:46,072 INFO crawl.CrawlDb - CrawlDb update: finished at
2013-04-21 16:04:46, elapsed: 00:00:13
2013-04-21 16:04:46,077 INFO crawl.Generator - Generator: starting at
2013-04-21 16:04:46
2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: filtering: true
2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: normalizing: true
2013-04-21 16:04:46,078 INFO crawl.Generator - Generator: topN: 5
2013-04-21 16:04:46,079 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2013-04-21 16:04:48,690 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:04:48,690 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:04:48,690 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:04:48,691 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:04:49,172 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:04:49,172 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:04:49,172 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:04:49,174 INFO regex.RegexURLNormalizer - can't find rules
for scope 'generate_host_count', using default
2013-04-21 16:04:52,685 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2013-04-21 16:04:53,686 INFO crawl.Generator - Generator: segment:
crawl/segments/20130421160453
2013-04-21 16:04:54,225 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:05:00,805 INFO crawl.Generator - Generator: finished at
2013-04-21 16:05:00, elapsed: 00:00:14
2013-04-21 16:05:00,805 WARN fetcher.Fetcher - Fetcher: Your
'http.agent.name' value should be listed first in 'http.robots.agents'
property.
2013-04-21 16:05:00,805 INFO fetcher.Fetcher - Fetcher: starting at
2013-04-21 16:05:00
2013-04-21 16:05:00,805 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20130421160453
2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Fetcher: threads: 10
2013-04-21 16:05:00,999 INFO fetcher.Fetcher - Fetcher: time-out divisor: 2
2013-04-21 16:05:01,006 INFO fetcher.Fetcher - QueueFeeder finished: total
5 records + hit by time limit :0
2013-04-21 16:05:01,021 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,022 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,023 INFO fetcher.Fetcher - fetching http://m.163.com/
2013-04-21 16:05:01,023 INFO http.Http - http.proxy.host = null
2013-04-21 16:05:01,030 INFO http.Http - http.proxy.port = 8080
2013-04-21 16:05:01,030 INFO http.Http - http.timeout = 10000
2013-04-21 16:05:01,030 INFO http.Http - http.content.limit = 65536
2013-04-21 16:05:01,030 INFO http.Http - http.agent = myagent/Nutch-1.6
2013-04-21 16:05:01,030 INFO http.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2013-04-21 16:05:01,030 INFO http.Http - http.accept =
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2013-04-21 16:05:01,024 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,055 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,056 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,057 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,063 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,064 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,064 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,071 INFO fetcher.Fetcher - fetching
http://music.163.com/
2013-04-21 16:05:01,069 INFO fetcher.Fetcher - fetching
http://caipiao.163.com/mobile/client_cp.jsp
2013-04-21 16:05:01,068 INFO fetcher.Fetcher - fetching
http://3g.163.com/links/4145
2013-04-21 16:05:01,087 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:01,088 INFO fetcher.Fetcher - Fetcher: throughput
threshold: -1
2013-04-21 16:05:01,089 INFO fetcher.Fetcher - Fetcher: throughput
threshold retries: 5
2013-04-21 16:05:01,265 INFO regex.RegexURLNormalizer - can't find rules
for scope 'fetcher', using default
2013-04-21 16:05:01,409 INFO regex.RegexURLNormalizer - can't find rules
for scope 'fetcher', using default
2013-04-21 16:05:02,090 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 16:05:02,090 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - nextFetchTime =
1366531506253
2013-04-21 16:05:02,091 INFO fetcher.Fetcher - now =
1366531502091
2013-04-21 16:05:02,092 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 16:05:03,092 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - nextFetchTime =
1366531506253
2013-04-21 16:05:03,093 INFO fetcher.Fetcher - now =
1366531503093
2013-04-21 16:05:03,094 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 16:05:04,094 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 16:05:04,095 INFO fetcher.Fetcher - nextFetchTime =
1366531506253
2013-04-21 16:05:04,095 INFO fetcher.Fetcher - now =
1366531504095
2013-04-21 16:05:04,095 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 16:05:05,096 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 16:05:05,096 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 16:05:05,096 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 16:05:05,096 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 16:05:05,097 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 16:05:05,097 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 16:05:05,097 INFO fetcher.Fetcher - nextFetchTime =
1366531506253
2013-04-21 16:05:05,097 INFO fetcher.Fetcher - now =
1366531505097
2013-04-21 16:05:05,097 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 16:05:06,098 INFO fetcher.Fetcher - -activeThreads=10,
spinWaiting=10, fetchQueues.totalSize=1
2013-04-21 16:05:06,098 INFO fetcher.Fetcher - * queue: http://m.163.com
2013-04-21 16:05:06,098 INFO fetcher.Fetcher - maxThreads = 1
2013-04-21 16:05:06,098 INFO fetcher.Fetcher - inProgress = 0
2013-04-21 16:05:06,098 INFO fetcher.Fetcher - crawlDelay = 5000
2013-04-21 16:05:06,099 INFO fetcher.Fetcher - minCrawlDelay = 0
2013-04-21 16:05:06,099 INFO fetcher.Fetcher - nextFetchTime =
1366531506253
2013-04-21 16:05:06,099 INFO fetcher.Fetcher - now =
1366531506099
2013-04-21 16:05:06,099 INFO fetcher.Fetcher - 0.
http://m.163.com/newsapp/
2013-04-21 16:05:06,264 INFO fetcher.Fetcher - fetching
http://m.163.com/newsapp/
2013-04-21 16:05:06,399 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=9
2013-04-21 16:05:06,434 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=8
2013-04-21 16:05:06,511 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=7
2013-04-21 16:05:06,573 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 16:05:06,573 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 16:05:06,574 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=4
2013-04-21 16:05:06,575 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=3
2013-04-21 16:05:06,590 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2013-04-21 16:05:06,591 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:05:06,722 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2013-04-21 16:05:07,100 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 16:05:07,100 INFO fetcher.Fetcher - -activeThreads=0
2013-04-21 16:05:13,874 INFO fetcher.Fetcher - Fetcher: finished at
2013-04-21 16:05:13, elapsed: 00:00:13
2013-04-21 16:05:13,874 INFO parse.ParseSegment - ParseSegment: starting at
2013-04-21 16:05:13
2013-04-21 16:05:13,875 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20130421160453
2013-04-21 16:05:14,289 INFO crawl.SignatureFactory - Using Signature impl:
org.apache.nutch.crawl.MD5Signature
2013-04-21 16:05:14,293 INFO parse.ParseSegment - Parsed
(4ms):http://caipiao.163.com/mobile/client_cp.jsp
2013-04-21 16:05:14,421 INFO parse.ParseSegment - Parsed
(4ms):http://m.163.com/newsapp/
2013-04-21 16:05:14,479 INFO parse.ParseSegment - Parsed
(1ms):http://music.163.com/
2013-04-21 16:05:16,960 INFO regex.RegexURLNormalizer - can't find rules
for scope 'outlink', using default
2013-04-21 16:05:20,928 INFO parse.ParseSegment - ParseSegment: finished at
2013-04-21 16:05:20, elapsed: 00:00:07
2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: starting at
2013-04-21 16:05:20
2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2013-04-21 16:05:20,934 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20130421160453]
2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2013-04-21 16:05:20,935 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
false
2013-04-21 16:05:20,936 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2013-04-21 16:05:21,108 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:05:24,107 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:05:27,120 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:05:30,025 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:05:30,025 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:05:30,025 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:05:34,015 INFO crawl.CrawlDb - CrawlDb update: finished at
2013-04-21 16:05:34, elapsed: 00:00:13
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: starting at
2013-04-21 16:05:34
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: filtering: true
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: normalizing: true
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: topN: 5
2013-04-21 16:05:34,018 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2013-04-21 16:05:34,363 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:05:34,363 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:05:34,363 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:05:34,364 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:05:37,077 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:05:37,077 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:05:37,077 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:05:37,078 INFO regex.RegexURLNormalizer - can't find rules
for scope 'generate_host_count', using default
2013-04-21 16:05:41,064 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2013-04-21 16:05:42,065 INFO crawl.Generator - Generator: segment:
crawl/segments/20130421160542
2013-04-21 16:05:42,232 INFO regex.RegexURLNormalizer - can't find rules
for scope 'partition', using default
2013-04-21 16:05:49,119 INFO crawl.Generator - Generator: finished at
2013-04-21 16:05:49, elapsed: 00:00:15
2013-04-21 16:05:49,120 WARN fetcher.Fetcher - Fetcher: Your
'http.agent.name' value should be listed first in 'http.robots.agents'
property.
2013-04-21 16:05:49,120 INFO fetcher.Fetcher - Fetcher: starting at
2013-04-21 16:05:49
2013-04-21 16:05:49,120 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20130421160542
2013-04-21 16:05:49,303 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,304 INFO fetcher.Fetcher - Fetcher: threads: 10
2013-04-21 16:05:49,304 INFO fetcher.Fetcher - Fetcher: time-out divisor: 2
2013-04-21 16:05:49,312 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,314 INFO fetcher.Fetcher - QueueFeeder finished: total
5 records + hit by time limit :0
2013-04-21 16:05:49,320 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,320 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,321 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,325 INFO fetcher.Fetcher - fetching
http://help.163.com/
2013-04-21 16:05:49,326 INFO http.Http - http.proxy.host = null
2013-04-21 16:05:49,326 INFO http.Http - http.proxy.port = 8080
2013-04-21 16:05:49,326 INFO http.Http - http.timeout = 10000
2013-04-21 16:05:49,326 INFO http.Http - http.content.limit = 65536
2013-04-21 16:05:49,326 INFO http.Http - http.agent = myagent/Nutch-1.6
2013-04-21 16:05:49,326 INFO http.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2013-04-21 16:05:49,326 INFO http.Http - http.accept =
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
2013-04-21 16:05:49,331 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,332 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,332 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,333 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,334 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,335 INFO fetcher.Fetcher - Using queue mode : byHost
2013-04-21 16:05:49,339 INFO fetcher.Fetcher - fetching
http://yuedu.163.com/
2013-04-21 16:05:49,340 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 16:05:49,340 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=6
2013-04-21 16:05:49,339 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=7
2013-04-21 16:05:49,341 INFO fetcher.Fetcher - Fetcher: throughput
threshold: -1
2013-04-21 16:05:49,341 INFO fetcher.Fetcher - Fetcher: throughput
threshold retries: 5
2013-04-21 16:05:49,341 INFO fetcher.Fetcher - fetching
http://open.163.com/
2013-04-21 16:05:49,341 INFO fetcher.Fetcher - fetching http://reg.163.com/
2013-04-21 16:05:49,342 INFO fetcher.Fetcher - fetching
http://sitemap.163.com/
2013-04-21 16:05:49,343 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 16:05:49,342 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=5
2013-04-21 16:05:49,407 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=4
2013-04-21 16:05:49,533 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=3
2013-04-21 16:05:49,568 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2013-04-21 16:05:49,737 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2013-04-21 16:05:49,943 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2013-04-21 16:05:50,341 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2013-04-21 16:05:50,342 INFO fetcher.Fetcher - -activeThreads=0
2013-04-21 16:05:56,181 INFO fetcher.Fetcher - Fetcher: finished at
2013-04-21 16:05:56, elapsed: 00:00:07
2013-04-21 16:05:56,181 INFO parse.ParseSegment - ParseSegment: starting at
2013-04-21 16:05:56
2013-04-21 16:05:56,182 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20130421160542
2013-04-21 16:05:56,489 INFO crawl.SignatureFactory - Using Signature impl:
org.apache.nutch.crawl.MD5Signature
2013-04-21 16:05:56,497 INFO parse.ParseSegment - Parsed
(8ms):http://help.163.com/
2013-04-21 16:05:56,627 INFO parse.ParseSegment - Parsed
(9ms):http://open.163.com/
2013-04-21 16:05:56,697 INFO parse.ParseSegment - Parsed
(3ms):http://reg.163.com/
2013-04-21 16:05:56,827 INFO parse.ParseSegment - Parsed
(6ms):http://sitemap.163.com/
2013-04-21 16:05:56,952 INFO parse.ParseSegment - Parsed
(4ms):http://yuedu.163.com/
2013-04-21 16:05:59,272 INFO regex.RegexURLNormalizer - can't find rules
for scope 'outlink', using default
2013-04-21 16:06:03,234 INFO parse.ParseSegment - ParseSegment: finished at
2013-04-21 16:06:03, elapsed: 00:00:07
2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: starting at
2013-04-21 16:06:03
2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2013-04-21 16:06:03,239 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20130421160542]
2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: 404 purging:
false
2013-04-21 16:06:03,240 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2013-04-21 16:06:03,419 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:06:06,450 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:06:09,408 INFO regex.RegexURLNormalizer - can't find rules
for scope 'crawldb', using default
2013-04-21 16:06:12,341 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2013-04-21 16:06:12,341 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2013-04-21 16:06:12,341 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2013-04-21 16:06:16,324 INFO crawl.CrawlDb - CrawlDb update: finished at
2013-04-21 16:06:16, elapsed: 00:00:13
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: starting at 2013-04-21
16:06:16
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: URL normalize: true
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: URL filter: true
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: internal links will be
ignored.
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421160453
2013-04-21 16:06:16,330 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421160408
2013-04-21 16:06:16,331 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/lailx/search_engine/nutch/crawl/segments/20130421160542
2013-04-21 16:06:16,483 INFO regex.RegexURLNormalizer - can't find rules
for scope 'linkdb', using default
2013-04-21 16:06:19,464 INFO regex.RegexURLNormalizer - can't find rules
for scope 'linkdb', using default
2013-04-21 16:06:22,498 INFO regex.RegexURLNormalizer - can't find rules
for scope 'linkdb', using default
2013-04-21 16:06:29,386 INFO crawl.LinkDb - LinkDb: finished at 2013-04-21
16:06:29, elapsed: 00:00:13
2013-04-21 16:06:29,387 INFO crawl.Crawl - crawl finished: crawl
2013-04-21 16:06:57,720 INFO solr.SolrIndexer - SolrIndexer: starting at
2013-04-21 16:06:57
2013-04-21 16:06:57,892 INFO solr.SolrIndexer - SolrIndexer: deleting gone
documents: false
2013-04-21 16:06:57,893 INFO solr.SolrIndexer - SolrIndexer: URL filtering:
false
2013-04-21 16:06:57,893 INFO solr.SolrIndexer - SolrIndexer: URL
normalizing: false
2013-04-21 16:06:57,915 INFO indexer.IndexerMapReduce - IndexerMapReduce:
crawldb: crawl/crawldb
2013-04-21 16:06:57,916 INFO indexer.IndexerMapReduce - IndexerMapReduce:
linkdb: crawl/linkdb
2013-04-21 16:06:57,916 INFO indexer.IndexerMapReduce - IndexerMapReduces:
adding segment: crawl/segments/20130421160408
2013-04-21 16:06:58,661 INFO indexer.IndexerMapReduce - IndexerMapReduces:
adding segment: crawl/segments/20130421160453
2013-04-21 16:06:58,663 INFO indexer.IndexerMapReduce - IndexerMapReduces:
adding segment: crawl/segments/20130421160542
2013-04-21 16:06:58,940 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2013-04-21 16:07:00,267 INFO plugin.PluginRepository - Plugins: looking in:
/home/lailx/search_engine/nutch/plugins
2013-04-21 16:07:00,604 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Registered Plugins:
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - the nutch core
extension points (nutch-extensionpoints)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL
Normalizer (urlnormalizer-regex)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - CyberNeko HTML
Parser (lib-nekohtml)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Basic URL
Normalizer (urlnormalizer-basic)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Tika Parser Plug-in
(parse-tika)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Html Parse Plug-in
(parse-html)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Basic Indexing
Filter (index-basic)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Anchor Indexing
Filter (index-anchor)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - HTTP Framework
(lib-http)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL Filter
(urlfilter-regex)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Regex URL Filter
Framework (lib-regex-filter)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Pass-through URL
Normalizer (urlnormalizer-pass)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Http Protocol
Plug-in (protocol-http)
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Registered
Extension-Points:
2013-04-21 16:07:00,605 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch Protocol
(org.apache.nutch.protocol.Protocol)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch Segment Merge
Filter (org.apache.nutch.segment.SegmentMergeFilter)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch URL Filter
(org.apache.nutch.net.URLFilter)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch Indexing
Filter (org.apache.nutch.indexer.IndexingFilter)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - HTML Parse Filter
(org.apache.nutch.parse.HtmlParseFilter)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch Content
Parser (org.apache.nutch.parse.Parser)
2013-04-21 16:07:00,606 INFO plugin.PluginRepository - Nutch Scoring
(org.apache.nutch.scoring.ScoringFilter)
2013-04-21 16:07:00,616 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:00,619 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:00,620 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:02,976 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:02,976 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:02,976 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:05,985 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:05,985 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:05,985 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:09,014 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:09,014 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:09,014 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:12,065 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:12,065 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:12,065 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:15,006 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:15,010 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:15,010 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:18,034 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:18,034 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:18,034 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:21,031 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:21,031 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:21,031 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:24,043 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:24,043 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:24,043 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:27,058 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:27,058 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:27,058 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:30,189 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:30,189 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:30,189 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:33,063 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:33,064 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:33,064 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:36,099 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:36,099 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:36,099 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:39,087 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:39,088 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:39,088 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:42,029 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2013-04-21 16:07:42,029 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2013-04-21 16:07:42,029 INFO indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: content dest:
content
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: title dest:
title
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: host dest:
host
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: segment dest:
segment
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: boost dest:
boost
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: digest dest:
digest
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: tstamp dest:
tstamp
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: url dest: id
2013-04-21 16:07:42,142 INFO solr.SolrMappingReader - source: url dest: url
2013-04-21 16:07:42,308 INFO solr.SolrWriter - Indexing 9 documents
2013-04-21 16:07:43,367 WARN mapred.LocalJobRunner - job_local_0001
*org.apache.solr.common.SolrException: Bad Request
Bad Request
request: http://127.0.0.1:8080/solr/update?wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:142)
at
org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.close(ReduceTask.java:466)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
2013-04-21 16:07:43,699 ERROR solr.SolrIndexer - java.io.IOException: Job
failed! *
Is the 'Bad Request' caused the exception?
Waiting your reply.
Thanks in advance.
note: I used solr-4.2.0 this time, still not work.
Br,
Mick
--
View this message in context: http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620p4057688.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by kiran chitturi <ch...@gmail.com>.
Sorry, here is the link http://wiki.apache.org/nutch/NutchTutorial
On Sat, Apr 20, 2013 at 4:11 PM, kiran chitturi
<ch...@gmail.com>wrote:
> Also, please go through the tutorial here [1]. We updated it with more
> info on commands and everything
>
>
> [1]
>
>
> On Sat, Apr 20, 2013 at 3:00 PM, micklai <la...@gmail.com> wrote:
>
>> Hi,
>>
>> Env:
>> System ubuntu 12.04
>> Tomcat: 7.0.39
>> Solr: 3.6.2
>> Nutch: 1.6
>>
>> Try to deploy nutch 1.6 with solr 3.6.2 but failed with running the
>> command
>> below:
>> bin/nutch crawl urls -solr http://localhost:8080/solr/ -dir crawl -depth
>> 2
>> -threads 5 -topN 5
>>
>> For the details:
>> =============================================
>> crawl started in: crawl
>> rootUrlDir = urls
>> threads = 5
>> depth = 2
>> solrUrl=http://localhost:8080/solr/
>> topN = 5
>> Injector: starting at 2013-04-21 02:21:12
>> Injector: crawlDb: crawl/crawldb
>> Injector: urlDir: urls
>> Injector: Converting injected urls to crawl db entries.
>> Injector: total number of urls rejected by filters: 0
>> Injector: total number of urls injected after normalization and
>> filtering: 1
>> Injector: Merging injected urls into crawl db.
>> Injector: finished at 2013-04-21 02:21:27, elapsed: 00:00:14
>> Generator: starting at 2013-04-21 02:21:27
>> Generator: Selecting best-scoring urls due for fetch.
>> Generator: filtering: true
>> Generator: normalizing: true
>> Generator: topN: 5
>> Generator: jobtracker is 'local', generating exactly one partition.
>> Generator: Partitioning selected urls for politeness.
>> Generator: segment: crawl/segments/20130421022135
>> Generator: finished at 2013-04-21 02:21:42, elapsed: 00:00:15
>> Fetcher: Your 'http.agent.name' value should be listed first in
>> 'http.robots.agents' property.
>> Fetcher: starting at 2013-04-21 02:21:42
>> Fetcher: segment: crawl/segments/20130421022135
>> Using queue mode : byHost
>> Fetcher: threads: 5
>> Fetcher: time-out divisor: 2
>> QueueFeeder finished: total 1 records + hit by time limit :0
>> Using queue mode : byHost
>> Using queue mode : byHost
>> Using queue mode : byHost
>> Using queue mode : byHost
>> Using queue mode : byHost
>> Fetcher: throughput threshold: -1
>> Fetcher: throughput threshold retries: 5
>> fetching http://www.163.com/
>> -finishing thread FetcherThread, activeThreads=4
>> -finishing thread FetcherThread, activeThreads=3
>> -finishing thread FetcherThread, activeThreads=2
>> -finishing thread FetcherThread, activeThreads=1
>> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
>> -finishing thread FetcherThread, activeThreads=0
>> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
>> -activeThreads=0
>> Fetcher: finished at 2013-04-21 02:21:49, elapsed: 00:00:07
>> ParseSegment: starting at 2013-04-21 02:21:49
>> ParseSegment: segment: crawl/segments/20130421022135
>> Parsed (24ms):http://www.163.com/
>> ParseSegment: finished at 2013-04-21 02:21:56, elapsed: 00:00:07
>> CrawlDb update: starting at 2013-04-21 02:21:56
>> CrawlDb update: db: crawl/crawldb
>> CrawlDb update: segments: [crawl/segments/20130421022135]
>> CrawlDb update: additions allowed: true
>> CrawlDb update: URL normalizing: true
>> CrawlDb update: URL filtering: true
>> CrawlDb update: 404 purging: false
>> CrawlDb update: Merging segment data into db.
>> CrawlDb update: finished at 2013-04-21 02:22:09, elapsed: 00:00:13
>> Generator: starting at 2013-04-21 02:22:09
>> Generator: Selecting best-scoring urls due for fetch.
>> Generator: filtering: true
>> Generator: normalizing: true
>> Generator: topN: 5
>> Generator: jobtracker is 'local', generating exactly one partition.
>> Generator: Partitioning selected urls for politeness.
>> Generator: segment: crawl/segments/20130421022217
>> Generator: finished at 2013-04-21 02:22:25, elapsed: 00:00:15
>> Fetcher: Your 'http.agent.name' value should be listed first in
>> 'http.robots.agents' property.
>> Fetcher: starting at 2013-04-21 02:22:25
>> Fetcher: segment: crawl/segments/20130421022217
>> Using queue mode : byHost
>> Fetcher: threads: 5
>> Fetcher: time-out divisor: 2
>> QueueFeeder finished: total 5 records + hit by time limit :0
>> Using queue mode : byHost
>> Using queue mode : byHost
>> fetching http://m.163.com/
>> Using queue mode : byHost
>> Using queue mode : byHost
>> fetching http://3g.163.com/links/4145
>> fetching http://music.163.com/
>> Using queue mode : byHost
>> Fetcher: throughput threshold: -1
>> Fetcher: throughput threshold retries: 5
>> fetching http://caipiao.163.com/mobile/client_cp.jsp
>> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
>> * queue: http://m.163.com
>> maxThreads = 1
>> inProgress = 0
>> crawlDelay = 5000
>> minCrawlDelay = 0
>> nextFetchTime = 1366482150482
>> now = 1366482146346
>> 0. http://m.163.com/newsapp/
>> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
>> * queue: http://m.163.com
>> maxThreads = 1
>> inProgress = 0
>> crawlDelay = 5000
>> minCrawlDelay = 0
>> nextFetchTime = 1366482150482
>> now = 1366482147348
>> 0. http://m.163.com/newsapp/
>> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
>> * queue: http://m.163.com
>> maxThreads = 1
>> inProgress = 0
>> crawlDelay = 5000
>> minCrawlDelay = 0
>> nextFetchTime = 1366482150482
>> now = 1366482148350
>> 0. http://m.163.com/newsapp/
>> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
>> * queue: http://m.163.com
>> maxThreads = 1
>> inProgress = 0
>> crawlDelay = 5000
>> minCrawlDelay = 0
>> nextFetchTime = 1366482150482
>> now = 1366482149352
>> 0. http://m.163.com/newsapp/
>> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
>> * queue: http://m.163.com
>> maxThreads = 1
>> inProgress = 0
>> crawlDelay = 5000
>> minCrawlDelay = 0
>> nextFetchTime = 1366482150482
>> now = 1366482150354
>> 0. http://m.163.com/newsapp/
>> fetching http://m.163.com/newsapp/
>> -finishing <http://m.163.com/newsapp/-finishing> thread FetcherThread,
>> activeThreads=4
>> -finishing thread FetcherThread, activeThreads=3
>> -finishing thread FetcherThread, activeThreads=2
>> -finishing thread FetcherThread, activeThreads=1
>> -finishing thread FetcherThread, activeThreads=0
>> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
>> -activeThreads=0
>> Fetcher: finished at 2013-04-21 02:22:38, elapsed: 00:00:13
>> ParseSegment: starting at 2013-04-21 02:22:38
>> ParseSegment: segment: crawl/segments/20130421022217
>> Parsed (4ms):http://caipiao.163.com/mobile/client_cp.jsp
>> Parsed (3ms):http://m.163.com/newsapp/
>> Parsed (1ms):http://music.163.com/
>> ParseSegment: finished at 2013-04-21 02:22:45, elapsed: 00:00:07
>> CrawlDb update: starting at 2013-04-21 02:22:45
>> CrawlDb update: db: crawl/crawldb
>> CrawlDb update: segments: [crawl/segments/20130421022217]
>> CrawlDb update: additions allowed: true
>> CrawlDb update: URL normalizing: true
>> CrawlDb update: URL filtering: true
>> *CrawlDb update: 404 purging: false*
>> CrawlDb update: Merging segment data into db.
>> CrawlDb update: finished at 2013-04-21 02:22:58, elapsed: 00:00:13
>> LinkDb: starting at 2013-04-21 02:22:58
>> LinkDb: linkdb: crawl/linkdb
>> LinkDb: URL normalize: true
>> LinkDb: URL filter: true
>> LinkDb: internal links will be ignored.
>> LinkDb: adding segment:
>> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022135
>> LinkDb: adding segment:
>> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022217
>> LinkDb: finished at 2013-04-21 02:23:08, elapsed: 00:00:10
>> SolrIndexer: starting at 2013-04-21 02:23:08
>> *SolrIndexer: deleting gone documents: false
>> SolrIndexer: URL filtering: false
>> SolrIndexer: URL normalizing: false*
>> Indexing 4 documents
>> *java.io.IOException: Job failed!*
>> SolrDeleteDuplicates: starting at 2013-04-21 02:23:39
>> SolrDeleteDuplicates: Solr url: http://localhost:8080/solr/
>> *Exception in thread "main" java.io.IOException: Job failed!*
>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
>> at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
>> at
>>
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
>> at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>> =============================================
>> But nutch works well, it works with the command below:
>> bin/nutch crawl urls -dir crawl -depth 2 -topN 5
>>
>> And I also had modified the [Solr_Home]/conf/schema.xml to make nutch
>> integrated with solr, solr also works well by accessing
>> "localhost:8080/solr".
>>
>> Hope you can help in this problem, wait for your reply.
>> Thanks.
>>
>> Br,
>> Mick
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620.html
>> Sent from the Nutch - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> Kiran Chitturi
>
> <http://www.linkedin.com/in/kiranchitturi>
>
>
>
--
Kiran Chitturi
<http://www.linkedin.com/in/kiranchitturi>
Re: [Exception in thread "main" java.io.IOException: Job failed!]
Posted by kiran chitturi <ch...@gmail.com>.
Also, please go through the tutorial here [1]. We updated it with more info
on commands and everything
[1]
On Sat, Apr 20, 2013 at 3:00 PM, micklai <la...@gmail.com> wrote:
> Hi,
>
> Env:
> System ubuntu 12.04
> Tomcat: 7.0.39
> Solr: 3.6.2
> Nutch: 1.6
>
> Try to deploy nutch 1.6 with solr 3.6.2 but failed with running the command
> below:
> bin/nutch crawl urls -solr http://localhost:8080/solr/ -dir crawl -depth 2
> -threads 5 -topN 5
>
> For the details:
> =============================================
> crawl started in: crawl
> rootUrlDir = urls
> threads = 5
> depth = 2
> solrUrl=http://localhost:8080/solr/
> topN = 5
> Injector: starting at 2013-04-21 02:21:12
> Injector: crawlDb: crawl/crawldb
> Injector: urlDir: urls
> Injector: Converting injected urls to crawl db entries.
> Injector: total number of urls rejected by filters: 0
> Injector: total number of urls injected after normalization and filtering:
> 1
> Injector: Merging injected urls into crawl db.
> Injector: finished at 2013-04-21 02:21:27, elapsed: 00:00:14
> Generator: starting at 2013-04-21 02:21:27
> Generator: Selecting best-scoring urls due for fetch.
> Generator: filtering: true
> Generator: normalizing: true
> Generator: topN: 5
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls for politeness.
> Generator: segment: crawl/segments/20130421022135
> Generator: finished at 2013-04-21 02:21:42, elapsed: 00:00:15
> Fetcher: Your 'http.agent.name' value should be listed first in
> 'http.robots.agents' property.
> Fetcher: starting at 2013-04-21 02:21:42
> Fetcher: segment: crawl/segments/20130421022135
> Using queue mode : byHost
> Fetcher: threads: 5
> Fetcher: time-out divisor: 2
> QueueFeeder finished: total 1 records + hit by time limit :0
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Using queue mode : byHost
> Fetcher: throughput threshold: -1
> Fetcher: throughput threshold retries: 5
> fetching http://www.163.com/
> -finishing thread FetcherThread, activeThreads=4
> -finishing thread FetcherThread, activeThreads=3
> -finishing thread FetcherThread, activeThreads=2
> -finishing thread FetcherThread, activeThreads=1
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> -finishing thread FetcherThread, activeThreads=0
> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> -activeThreads=0
> Fetcher: finished at 2013-04-21 02:21:49, elapsed: 00:00:07
> ParseSegment: starting at 2013-04-21 02:21:49
> ParseSegment: segment: crawl/segments/20130421022135
> Parsed (24ms):http://www.163.com/
> ParseSegment: finished at 2013-04-21 02:21:56, elapsed: 00:00:07
> CrawlDb update: starting at 2013-04-21 02:21:56
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20130421022135]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> CrawlDb update: 404 purging: false
> CrawlDb update: Merging segment data into db.
> CrawlDb update: finished at 2013-04-21 02:22:09, elapsed: 00:00:13
> Generator: starting at 2013-04-21 02:22:09
> Generator: Selecting best-scoring urls due for fetch.
> Generator: filtering: true
> Generator: normalizing: true
> Generator: topN: 5
> Generator: jobtracker is 'local', generating exactly one partition.
> Generator: Partitioning selected urls for politeness.
> Generator: segment: crawl/segments/20130421022217
> Generator: finished at 2013-04-21 02:22:25, elapsed: 00:00:15
> Fetcher: Your 'http.agent.name' value should be listed first in
> 'http.robots.agents' property.
> Fetcher: starting at 2013-04-21 02:22:25
> Fetcher: segment: crawl/segments/20130421022217
> Using queue mode : byHost
> Fetcher: threads: 5
> Fetcher: time-out divisor: 2
> QueueFeeder finished: total 5 records + hit by time limit :0
> Using queue mode : byHost
> Using queue mode : byHost
> fetching http://m.163.com/
> Using queue mode : byHost
> Using queue mode : byHost
> fetching http://3g.163.com/links/4145
> fetching http://music.163.com/
> Using queue mode : byHost
> Fetcher: throughput threshold: -1
> Fetcher: throughput threshold retries: 5
> fetching http://caipiao.163.com/mobile/client_cp.jsp
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482146346
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482147348
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482148350
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482149352
> 0. http://m.163.com/newsapp/
> -activeThreads=5, spinWaiting=4, fetchQueues.totalSize=1
> * queue: http://m.163.com
> maxThreads = 1
> inProgress = 0
> crawlDelay = 5000
> minCrawlDelay = 0
> nextFetchTime = 1366482150482
> now = 1366482150354
> 0. http://m.163.com/newsapp/
> fetching http://m.163.com/newsapp/
> -finishing thread FetcherThread, activeThreads=4
> -finishing thread FetcherThread, activeThreads=3
> -finishing thread FetcherThread, activeThreads=2
> -finishing thread FetcherThread, activeThreads=1
> -finishing thread FetcherThread, activeThreads=0
> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> -activeThreads=0
> Fetcher: finished at 2013-04-21 02:22:38, elapsed: 00:00:13
> ParseSegment: starting at 2013-04-21 02:22:38
> ParseSegment: segment: crawl/segments/20130421022217
> Parsed (4ms):http://caipiao.163.com/mobile/client_cp.jsp
> Parsed (3ms):http://m.163.com/newsapp/
> Parsed (1ms):http://music.163.com/
> ParseSegment: finished at 2013-04-21 02:22:45, elapsed: 00:00:07
> CrawlDb update: starting at 2013-04-21 02:22:45
> CrawlDb update: db: crawl/crawldb
> CrawlDb update: segments: [crawl/segments/20130421022217]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> *CrawlDb update: 404 purging: false*
> CrawlDb update: Merging segment data into db.
> CrawlDb update: finished at 2013-04-21 02:22:58, elapsed: 00:00:13
> LinkDb: starting at 2013-04-21 02:22:58
> LinkDb: linkdb: crawl/linkdb
> LinkDb: URL normalize: true
> LinkDb: URL filter: true
> LinkDb: internal links will be ignored.
> LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022135
> LinkDb: adding segment:
> file:/home/lailx/search_engine/nutch/crawl/segments/20130421022217
> LinkDb: finished at 2013-04-21 02:23:08, elapsed: 00:00:10
> SolrIndexer: starting at 2013-04-21 02:23:08
> *SolrIndexer: deleting gone documents: false
> SolrIndexer: URL filtering: false
> SolrIndexer: URL normalizing: false*
> Indexing 4 documents
> *java.io.IOException: Job failed!*
> SolrDeleteDuplicates: starting at 2013-04-21 02:23:39
> SolrDeleteDuplicates: Solr url: http://localhost:8080/solr/
> *Exception in thread "main" java.io.IOException: Job failed!*
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
> at
>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:373)
> at
>
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:353)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> =============================================
> But nutch works well, it works with the command below:
> bin/nutch crawl urls -dir crawl -depth 2 -topN 5
>
> And I also had modified the [Solr_Home]/conf/schema.xml to make nutch
> integrated with solr, solr also works well by accessing
> "localhost:8080/solr".
>
> Hope you can help in this problem, wait for your reply.
> Thanks.
>
> Br,
> Mick
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Exception-in-thread-main-java-io-IOException-Job-failed-tp4057620.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
Kiran Chitturi
<http://www.linkedin.com/in/kiranchitturi>