You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Carl Zha <ca...@theportalgrp.com> on 2011/02/17 00:04:27 UTC
nutch ntlm authentication failing in nutch 1.2
Hi,
We are using nutch 1.2
we have overridden the 'plugin.includes' property of
'conf/nutch-default.xml' with 'conf/nutch-site.xml' and replaced
'protocol-http' with 'protocol-httpclient'.
please see attached nutch_site.xml
we configured 'conf/httpclient-auth.xml'
see attached
have enabled debug logging for 'protocol-httpclient' and HttpClient, open
'conf/log4j.properties' and add the following lines:
log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
two seed urls are http://tfs/ and http://tv/
authentication is failing, Please help!
Following is the log output:
2011-02-16 14:32:03,755 INFO crawl.Injector - Injector: starting at
2011-02-16 14:32:03
2011-02-16 14:32:03,756 INFO crawl.Injector - Injector: crawlDb: crawl/crawldb
2011-02-16 14:32:03,757 INFO crawl.Injector - Injector: urlDir: urls
2011-02-16 14:32:03,758 INFO crawl.Injector - Injector: Converting
injected urls to crawl db entries.
2011-02-16 14:32:06,314 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:07,863 INFO crawl.Injector - Injector: Merging
injected urls into crawl db.
2011-02-16 14:32:09,857 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2011-02-16 14:32:10,376 INFO crawl.Injector - Injector: finished at
2011-02-16 14:32:10, elapsed: 00:00:06
2011-02-16 14:32:12,218 INFO crawl.Generator - Generator: starting at
2011-02-16 14:32:12
2011-02-16 14:32:12,219 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2011-02-16 14:32:12,220 INFO crawl.Generator - Generator: filtering: true
2011-02-16 14:32:12,220 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 14:32:12,224 INFO crawl.Generator - Generator: jobtracker
is 'local', generating exactly one partition.
2011-02-16 14:32:14,409 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:14,809 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:14,810 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 14:32:14,810 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:15,003 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:15,240 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:15,241 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 14:32:15,241 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:16,040 INFO crawl.Generator - Generator:
Partitioning selected urls for politeness.
2011-02-16 14:32:17,043 INFO crawl.Generator - Generator: segment:
crawl/segments/20110216143217
2011-02-16 14:32:18,662 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:18,872 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:19,454 INFO crawl.Generator - Generator: finished at
2011-02-16 14:32:19, elapsed: 00:00:07
2011-02-16 14:32:20,955 INFO fetcher.Fetcher - Fetcher: starting at
2011-02-16 14:32:20
2011-02-16 14:32:20,956 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20110216143217
2011-02-16 14:32:23,347 INFO fetcher.Fetcher - Fetcher: threads: 10
2011-02-16 14:32:23,374 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:23,374 INFO fetcher.Fetcher - QueueFeeder finished:
total 2 records + hit by time limit :0
2011-02-16 14:32:23,741 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:23,782 INFO fetcher.Fetcher - fetching http://tv/
2011-02-16 14:32:23,782 INFO fetcher.Fetcher - fetching http://tfs/
2011-02-16 14:32:23,783 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,783 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,784 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,784 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=3
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 14:32:23,839 INFO httpclient.Http - http.proxy.host =
proxy.services.toyota.com
2011-02-16 14:32:23,839 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 14:32:23,839 INFO httpclient.Http - http.timeout = 10000
2011-02-16 14:32:23,839 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 14:32:23,840 INFO httpclient.Http - http.agent =
nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch
1.2; http://devcpd1.lexus.com/; carl.zha@theportgrp.com)
2011-02-16 14:32:23,840 INFO httpclient.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 14:32:23,840 INFO httpclient.Http -
protocol.plugin.check.blocking = false
2011-02-16 14:32:23,840 INFO httpclient.Http -
protocol.plugin.check.robots = false
2011-02-16 14:32:23,853 INFO httpclient.Http - http.proxy.host =
proxy.services.toyota.com
2011-02-16 14:32:23,854 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 14:32:23,854 INFO httpclient.Http - http.timeout = 10000
2011-02-16 14:32:23,854 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 14:32:23,854 INFO httpclient.Http - http.agent =
nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch
1.2; http://devcpd1.lexus.com/; carl.zha@theportgrp.com)
2011-02-16 14:32:23,854 INFO httpclient.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 14:32:23,854 INFO httpclient.Http -
protocol.plugin.check.blocking = false
2011-02-16 14:32:23,855 INFO httpclient.Http -
protocol.plugin.check.robots = false
2011-02-16 14:32:24,016 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest,
basic]
2011-02-16 14:32:24,017 INFO auth.AuthChallengeProcessor - ntlm
authentication scheme selected
2011-02-16 14:32:24,017 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:24,017 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:24,111 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:24,111 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:24,791 INFO fetcher.Fetcher - -activeThreads=2,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:24,971 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:24,971 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:24,971 INFO httpclient.HttpMethodDirector - Failure
authenticating with NTLM <any realm>@tfs:80
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest,
basic]
2011-02-16 14:32:25,054 INFO auth.AuthChallengeProcessor - ntlm
authentication scheme selected
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:25,127 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:25,127 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:25,193 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 14:32:25,193 DEBUG auth.AuthChallengeProcessor -
Authorization challenge processed
2011-02-16 14:32:25,193 INFO httpclient.HttpMethodDirector - Failure
authenticating with NTLM <any realm>@tfs:80
2011-02-16 14:32:25,497 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2011-02-16 14:32:25,801 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:26,811 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:27,821 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:28,831 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:29,841 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:30,851 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:31,861 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:32,871 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:33,881 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:33,988 INFO api.RobotRulesParser - Couldn't get
robots.txt for http://tv/: java.net.SocketTimeoutException: Read timed
out
2011-02-16 14:32:34,891 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:35,901 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:36,910 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:37,921 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:38,931 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:39,940 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:40,950 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:41,960 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:42,970 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:43,980 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:44,004 ERROR httpclient.Http -
java.net.SocketTimeoutException: Read timed out
2011-02-16 14:32:44,005 ERROR httpclient.Http - at
java.net.SocketInputStream.socketRead0(Native Method)
2011-02-16 14:32:44,005 ERROR httpclient.Http - at
java.net.SocketInputStream.read(SocketInputStream.java:129)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
2011-02-16 14:32:44,008 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at
org.apache.nutch.protocol.httpclient.HttpResponse.<init>(HttpResponse.java:94)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at
org.apache.nutch.protocol.httpclient.Http.getResponse(Http.java:154)
2011-02-16 14:32:44,011 ERROR httpclient.Http - at
org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:224)
2011-02-16 14:32:44,011 ERROR httpclient.Http - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:628)
2011-02-16 14:32:44,011 INFO fetcher.Fetcher - fetch of http://tv/
failed with: java.net.SocketTimeoutException: Read timed out
2011-02-16 14:32:44,012 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2011-02-16 14:32:44,990 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:44,990 INFO fetcher.Fetcher - -activeThreads=0
2011-02-16 14:32:45,234 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2011-02-16 14:32:46,236 INFO fetcher.Fetcher - Fetcher: finished at
2011-02-16 14:32:46, elapsed: 00:00:25
2011-02-16 14:32:47,763 INFO parse.ParseSegment - ParseSegment:
starting at 2011-02-16 14:32:47
2011-02-16 14:32:47,764 INFO parse.ParseSegment - ParseSegment:
segment: crawl/segments/20110216143217
2011-02-16 14:32:49,938 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2011-02-16 14:32:50,194 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:50,566 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:50,742 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:51,046 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:51,046 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:51,936 INFO parse.ParseSegment - ParseSegment:
finished at 2011-02-16 14:32:51, elapsed: 00:00:04
2011-02-16 14:32:53,722 INFO crawl.CrawlDb - CrawlDb update: starting
at 2011-02-16 14:32:53
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update: db: crawl/crawldb
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update:
segments: [crawl/segments/20110216143217]
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update:
additions allowed: true
2011-02-16 14:32:53,724 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2011-02-16 14:32:53,724 INFO crawl.CrawlDb - CrawlDb update: URL
filtering: true
2011-02-16 14:32:53,753 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2011-02-16 14:32:55,862 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:56,244 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:56,244 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:56,468 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2011-02-16 14:32:56,677 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,407 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,677 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,876 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:57,877 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 14:32:57,877 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:58,526 INFO crawl.CrawlDb - CrawlDb update: finished
at 2011-02-16 14:32:58, elapsed: 00:00:04
2011-02-16 14:33:00,393 INFO crawl.Generator - Generator: starting at
2011-02-16 14:33:00
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: filtering: true
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 14:33:00,395 INFO crawl.Generator - Generator: topN: 1000
2011-02-16 14:33:00,398 INFO crawl.Generator - Generator: jobtracker
is 'local', generating exactly one partition.
2011-02-16 14:33:02,568 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:33:02,952 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:33:02,975 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:33:02,976 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 14:33:02,977 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:33:03,168 INFO plugin.PluginRepository - Plugins:
looking in: /www/nutch-1.2/plugins
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - the
nutch core extension points (nutch-extensionpoints)
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Basic
Query Filter (query-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Html
Parse Plug-in (parse-html)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Http /
Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Site
Query Filter (query-site)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Regex
URL Filter (urlfilter-regex)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - XML
Response Writer Plug-in (response-xml)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - OPIC
Scoring Plug-in (scoring-opic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository -
CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - URL
Query Filter (query-url)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Regex
URL Filter Framework (lib-regex-filter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - HTML
Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
Search Results Response Writer
(org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch
URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch
URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch
Online Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository -
Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:33:03,398 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:33:03,398 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 14:33:03,398 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:33:04,190 WARN crawl.Generator - Generator: 0 records
selected for fetching, exiting ...
2011-02-16 14:33:06,042 INFO crawl.LinkDb - LinkDb: starting at
2011-02-16 14:33:06
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: URL normalize: true
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: URL filter: true
2011-02-16 14:33:06,061 INFO crawl.LinkDb - LinkDb: adding segment:
file:/www/nutch-1.2/crawl/segments/20110216143217
2011-02-16 14:33:07,941 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2011-02-16 14:33:08,831 INFO crawl.LinkDb - LinkDb: finished at
2011-02-16 14:33:08, elapsed: 00:00:02
Re: nutch ntlm authentication failing in nutch 1.2
Posted by Carl Zha <ca...@theportalgrp.com>.
log output after configuring the realm. didn't look different from before:
2011-02-16 15:23:53,405 INFO crawl.Injector - Injector: starting at
2011-02-16 15:23:53
2011-02-16 15:23:53,406 INFO crawl.Injector - Injector: crawlDb:
crawl/crawldb
2011-02-16 15:23:53,406 INFO crawl.Injector - Injector: urlDir: urls
2011-02-16 15:23:53,408 INFO crawl.Injector - Injector: Converting injected
urls to crawl db entries.
2011-02-16 15:23:56,007 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:23:56,392 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:23:56,392 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:23:56,392 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:23:56,392 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:23:56,393 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:23:56,393 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:23:56,397 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:23:56,397 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:23:56,397 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:23:56,398 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:23:56,399 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:23:56,400 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:23:56,400 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:23:56,400 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:23:57,530 INFO crawl.Injector - Injector: Merging injected
urls into crawl db.
2011-02-16 15:23:59,553 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2011-02-16 15:24:00,111 INFO crawl.Injector - Injector: finished at
2011-02-16 15:24:00, elapsed: 00:00:06
2011-02-16 15:24:02,016 INFO crawl.Generator - Generator: starting at
2011-02-16 15:24:02
2011-02-16 15:24:02,017 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2011-02-16 15:24:02,017 INFO crawl.Generator - Generator: filtering: true
2011-02-16 15:24:02,017 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 15:24:02,021 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2011-02-16 15:24:04,104 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:04,480 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:04,480 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:04,481 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:04,482 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:04,483 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:04,508 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 15:24:04,509 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 15:24:04,509 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2011-02-16 15:24:04,696 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:04,907 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:04,908 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:04,909 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:04,910 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:04,911 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:04,924 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 15:24:04,925 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 15:24:04,925 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2011-02-16 15:24:05,738 INFO crawl.Generator - Generator: Partitioning
selected urls for politeness.
2011-02-16 15:24:06,739 INFO crawl.Generator - Generator: segment:
crawl/segments/20110216152406
2011-02-16 15:24:08,377 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:08,585 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:08,585 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:08,585 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:08,586 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:08,587 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:09,159 INFO crawl.Generator - Generator: finished at
2011-02-16 15:24:09, elapsed: 00:00:07
2011-02-16 15:24:10,685 INFO fetcher.Fetcher - Fetcher: starting at
2011-02-16 15:24:10
2011-02-16 15:24:10,686 INFO fetcher.Fetcher - Fetcher: segment:
crawl/segments/20110216152406
2011-02-16 15:24:13,154 INFO fetcher.Fetcher - Fetcher: threads: 10
2011-02-16 15:24:13,180 INFO fetcher.Fetcher - QueueFeeder finished: total
2 records + hit by time limit :0
2011-02-16 15:24:13,181 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:13,544 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:13,545 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:13,546 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:13,547 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:13,547 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:13,547 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:13,585 INFO fetcher.Fetcher - fetching http://tfs/
2011-02-16 15:24:13,585 INFO fetcher.Fetcher - fetching http://tv/
2011-02-16 15:24:13,586 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,586 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,585 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,586 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,587 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,587 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,587 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,587 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=2
2011-02-16 15:24:13,643 INFO httpclient.Http - http.proxy.host =
proxy.services.toyota.com
2011-02-16 15:24:13,643 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 15:24:13,643 INFO httpclient.Http - http.timeout = 10000
2011-02-16 15:24:13,643 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 15:24:13,644 INFO httpclient.Http - http.agent =
nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch 1.2;
http://devcpd1.lexus.com/; carl.zha@theportgrp.com)
2011-02-16 15:24:13,644 INFO httpclient.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 15:24:13,644 INFO httpclient.Http -
protocol.plugin.check.blocking = false
2011-02-16 15:24:13,644 INFO httpclient.Http - protocol.plugin.check.robots
= false
2011-02-16 15:24:13,658 INFO httpclient.Http - http.proxy.host =
proxy.services.toyota.com
2011-02-16 15:24:13,658 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 15:24:13,658 INFO httpclient.Http - http.timeout = 10000
2011-02-16 15:24:13,658 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 15:24:13,658 INFO httpclient.Http - http.agent =
nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch 1.2;
http://devcpd1.lexus.com/; carl.zha@theportgrp.com)
2011-02-16 15:24:13,659 INFO httpclient.Http - http.accept.language =
en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 15:24:13,659 INFO httpclient.Http -
protocol.plugin.check.blocking = false
2011-02-16 15:24:13,659 INFO httpclient.Http - protocol.plugin.check.robots
= false
2011-02-16 15:24:13,957 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest, basic]
2011-02-16 15:24:13,958 INFO auth.AuthChallengeProcessor - ntlm
authentication scheme selected
2011-02-16 15:24:13,958 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:13,958 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:14,022 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:14,022 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:14,588 INFO fetcher.Fetcher - -activeThreads=2,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:14,876 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:14,876 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:14,877 INFO httpclient.HttpMethodDirector - Failure
authenticating with NTLM <any realm>@tfs:80
2011-02-16 15:24:14,944 DEBUG auth.AuthChallengeProcessor - Supported
authentication schemes in the order of preference: [ntlm, digest, basic]
2011-02-16 15:24:14,944 INFO auth.AuthChallengeProcessor - ntlm
authentication scheme selected
2011-02-16 15:24:14,944 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:14,945 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:15,019 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:15,019 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:15,095 DEBUG auth.AuthChallengeProcessor - Using
authentication scheme: ntlm
2011-02-16 15:24:15,095 DEBUG auth.AuthChallengeProcessor - Authorization
challenge processed
2011-02-16 15:24:15,095 INFO httpclient.HttpMethodDirector - Failure
authenticating with NTLM <any realm>@tfs:80
2011-02-16 15:24:15,397 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=1
2011-02-16 15:24:15,598 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:16,608 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:17,618 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:18,628 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:19,638 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:20,648 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:21,657 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:22,667 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:23,677 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:23,768 INFO api.RobotRulesParser - Couldn't get robots.txt
for http://tv/: java.net.SocketTimeoutException: Read timed out
2011-02-16 15:24:24,687 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:25,697 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:26,707 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:27,717 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:28,727 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:29,737 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:30,747 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:31,757 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:32,767 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:33,777 INFO fetcher.Fetcher - -activeThreads=1,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:33,801 ERROR httpclient.Http -
java.net.SocketTimeoutException: Read timed out
2011-02-16 15:24:33,802 ERROR httpclient.Http - at
java.net.SocketInputStream.socketRead0(Native Method)
2011-02-16 15:24:33,802 ERROR httpclient.Http - at
java.net.SocketInputStream.read(SocketInputStream.java:129)
2011-02-16 15:24:33,802 ERROR httpclient.Http - at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
2011-02-16 15:24:33,802 ERROR httpclient.Http - at
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
2011-02-16 15:24:33,802 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
2011-02-16 15:24:33,803 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
2011-02-16 15:24:33,803 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
2011-02-16 15:24:33,803 ERROR httpclient.Http - at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
2011-02-16 15:24:33,803 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
2011-02-16 15:24:33,803 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
2011-02-16 15:24:33,804 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
2011-02-16 15:24:33,804 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
2011-02-16 15:24:33,804 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
2011-02-16 15:24:33,804 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
2011-02-16 15:24:33,806 ERROR httpclient.Http - at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
2011-02-16 15:24:33,807 ERROR httpclient.Http - at
org.apache.nutch.protocol.httpclient.HttpResponse.<init>(HttpResponse.java:94)
2011-02-16 15:24:33,807 ERROR httpclient.Http - at
org.apache.nutch.protocol.httpclient.Http.getResponse(Http.java:154)
2011-02-16 15:24:33,807 ERROR httpclient.Http - at
org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:224)
2011-02-16 15:24:33,807 ERROR httpclient.Http - at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:628)
2011-02-16 15:24:33,808 INFO fetcher.Fetcher - fetch of http://tv/ failed
with: java.net.SocketTimeoutException: Read timed out
2011-02-16 15:24:33,808 INFO fetcher.Fetcher - -finishing thread
FetcherThread, activeThreads=0
2011-02-16 15:24:34,787 INFO fetcher.Fetcher - -activeThreads=0,
spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 15:24:34,787 INFO fetcher.Fetcher - -activeThreads=0
2011-02-16 15:24:35,042 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2011-02-16 15:24:36,003 INFO fetcher.Fetcher - Fetcher: finished at
2011-02-16 15:24:36, elapsed: 00:00:25
2011-02-16 15:24:37,470 INFO parse.ParseSegment - ParseSegment: starting at
2011-02-16 15:24:37
2011-02-16 15:24:37,471 INFO parse.ParseSegment - ParseSegment: segment:
crawl/segments/20110216152406
2011-02-16 15:24:39,644 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2011-02-16 15:24:39,918 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:40,294 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:40,295 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:40,296 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:40,297 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:40,297 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:40,297 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:40,297 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:40,468 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:40,779 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:40,780 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:40,781 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:40,782 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:41,552 INFO parse.ParseSegment - ParseSegment: finished at
2011-02-16 15:24:41, elapsed: 00:00:04
2011-02-16 15:24:43,391 INFO crawl.CrawlDb - CrawlDb update: starting at
2011-02-16 15:24:43
2011-02-16 15:24:43,392 INFO crawl.CrawlDb - CrawlDb update: db:
crawl/crawldb
2011-02-16 15:24:43,392 INFO crawl.CrawlDb - CrawlDb update: segments:
[crawl/segments/20110216152406]
2011-02-16 15:24:43,392 INFO crawl.CrawlDb - CrawlDb update: additions
allowed: true
2011-02-16 15:24:43,392 INFO crawl.CrawlDb - CrawlDb update: URL
normalizing: true
2011-02-16 15:24:43,392 INFO crawl.CrawlDb - CrawlDb update: URL filtering:
true
2011-02-16 15:24:43,422 INFO crawl.CrawlDb - CrawlDb update: Merging
segment data into db.
2011-02-16 15:24:45,526 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:45,897 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:45,897 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:45,898 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:45,899 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:45,900 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:46,124 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2011-02-16 15:24:46,561 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:46,775 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:46,776 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:46,777 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:46,778 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:47,123 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:47,309 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:47,310 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:47,311 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:47,312 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:47,381 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:47,694 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:47,695 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:47,696 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:47,702 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 15:24:47,703 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 15:24:47,703 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2011-02-16 15:24:48,195 INFO crawl.CrawlDb - CrawlDb update: finished at
2011-02-16 15:24:48, elapsed: 00:00:04
2011-02-16 15:24:50,012 INFO crawl.Generator - Generator: starting at
2011-02-16 15:24:50
2011-02-16 15:24:50,013 INFO crawl.Generator - Generator: Selecting
best-scoring urls due for fetch.
2011-02-16 15:24:50,013 INFO crawl.Generator - Generator: filtering: true
2011-02-16 15:24:50,013 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 15:24:50,013 INFO crawl.Generator - Generator: topN: 1000
2011-02-16 15:24:50,016 INFO crawl.Generator - Generator: jobtracker is
'local', generating exactly one partition.
2011-02-16 15:24:52,179 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:52,554 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:52,554 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:52,554 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:52,554 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:52,554 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:52,555 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:52,556 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:52,557 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:52,582 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 15:24:52,583 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 15:24:52,583 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2011-02-16 15:24:52,762 INFO plugin.PluginRepository - Plugins: looking in:
/www/nutch-1.2/plugins
2011-02-16 15:24:52,978 INFO plugin.PluginRepository - Plugin
Auto-activation mode: [true]
2011-02-16 15:24:52,978 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 15:24:52,978 INFO plugin.PluginRepository - the nutch
core extension points (nutch-extensionpoints)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Basic Query
Filter (query-basic)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Basic
Indexing Filter (index-basic)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Html Parse
Plug-in (parse-html)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Basic
Summarizer Plug-in (summary-basic)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Site Query
Filter (query-site)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - HTTP
Framework (lib-http)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Regex URL
Filter (urlfilter-regex)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - XML Response
Writer Plug-in (response-xml)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - CyberNeko
HTML Parser (lib-nekohtml)
2011-02-16 15:24:52,979 INFO plugin.PluginRepository - Anchor
Indexing Filter (index-anchor)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - URL Query
Filter (query-url)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Regex URL
Filter Framework (lib-regex-filter)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - JSON
Response Writer Plug-in (response-json)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Registered
Extension-Points:
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch
Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch
Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch
Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch
Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch Field
Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - HTML Parse
Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 15:24:52,980 INFO plugin.PluginRepository - Nutch Query
Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch URL
Filter (org.apache.nutch.net.URLFilter)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch Online
Search Results Clustering Plugin
(org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch
Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch
Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Nutch
Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 15:24:52,981 INFO plugin.PluginRepository - Ontology
Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 15:24:52,995 INFO crawl.FetchScheduleFactory - Using
FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 15:24:52,995 INFO crawl.AbstractFetchSchedule -
defaultInterval=2592000
2011-02-16 15:24:52,996 INFO crawl.AbstractFetchSchedule -
maxInterval=7776000
2011-02-16 15:24:53,777 WARN crawl.Generator - Generator: 0 records
selected for fetching, exiting ...
2011-02-16 15:24:55,591 INFO crawl.LinkDb - LinkDb: starting at 2011-02-16
15:24:55
2011-02-16 15:24:55,592 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2011-02-16 15:24:55,592 INFO crawl.LinkDb - LinkDb: URL normalize: true
2011-02-16 15:24:55,592 INFO crawl.LinkDb - LinkDb: URL filter: true
2011-02-16 15:24:55,609 INFO crawl.LinkDb - LinkDb: adding segment:
file:/www/nutch-1.2/crawl/segments/20110216152406
2011-02-16 15:24:57,444 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2011-02-16 15:24:58,323 INFO crawl.LinkDb - LinkDb: finished at 2011-02-16
15:24:58, elapsed: 00:00:02
On Wed, Feb 16, 2011 at 3:29 PM, Carl Zha <ca...@theportalgrp.com> wrote:
> Thanks for the response, Arkadi. I have configured our realm by changing
> <default/> to <default realm="TMS" /> in conf/httpclient-auth.xml. But the
> crawl still no go.
>
> Regards,
>
> Carl
>
>
> On Wed, Feb 16, 2011 at 3:12 PM, <Ar...@csiro.au> wrote:
>
>> Hi Carl,
>>
>> Try configuring your realm:
>>
>> Change <default/> to <default realm="your_realm" />
>>
>> This works for me.
>>
>> Regards,
>>
>> Arkadi
>>
>>
Re: nutch ntlm authentication failing in nutch 1.2
Posted by Carl Zha <ca...@theportalgrp.com>.
Thanks for the response, Arkadi. I have configured our realm by changing
<default/> to <default realm="TMS" /> in conf/httpclient-auth.xml. But the
crawl still no go.
Regards,
Carl
On Wed, Feb 16, 2011 at 3:12 PM, <Ar...@csiro.au> wrote:
> Hi Carl,
>
> Try configuring your realm:
>
> Change <default/> to <default realm="your_realm" />
>
> This works for me.
>
> Regards,
>
> Arkadi
>
>
RE: nutch ntlm authentication failing in nutch 1.2
Posted by Ar...@csiro.au.
Hi Carl,
Try configuring your realm:
Change <default/> to <default realm="your_realm" />
This works for me.
Regards,
Arkadi
From: Carl Zha [mailto:carl.zha@theportalgrp.com]
Sent: Thursday, February 17, 2011 10:04 AM
To: user@nutch.apache.org
Subject: nutch ntlm authentication failing in nutch 1.2
Hi,
We are using nutch 1.2
we have overridden the 'plugin.includes' property of 'conf/nutch-default.xml' with 'conf/nutch-site.xml' and replaced 'protocol-http' with 'protocol-httpclient'.
please see attached nutch_site.xml
we configured 'conf/httpclient-auth.xml'
see attached
have enabled debug logging for 'protocol-httpclient' and HttpClient, open 'conf/log4j.properties' and add the following lines:
log4j.logger.org.apache.nutch.protocol.httpclient=DEBUG,cmdstdout
log4j.logger.org.apache.commons.httpclient.auth=DEBUG,cmdstdout
two seed urls are http://tfs/ and http://tv/
authentication is failing, Please help!
Following is the log output:
2011-02-16 14:32:03,755 INFO crawl.Injector - Injector: starting at 2011-02-16 14:32:03
2011-02-16 14:32:03,756 INFO crawl.Injector - Injector: crawlDb: crawl/crawldb
2011-02-16 14:32:03,757 INFO crawl.Injector - Injector: urlDir: urls
2011-02-16 14:32:03,758 INFO crawl.Injector - Injector: Converting injected urls to crawl db entries.
2011-02-16 14:32:06,314 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:06,684 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:06,685 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:06,686 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:06,687 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:07,863 INFO crawl.Injector - Injector: Merging injected urls into crawl db.
2011-02-16 14:32:09,857 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-02-16 14:32:10,376 INFO crawl.Injector - Injector: finished at 2011-02-16 14:32:10, elapsed: 00:00:06
2011-02-16 14:32:12,218 INFO crawl.Generator - Generator: starting at 2011-02-16 14:32:12
2011-02-16 14:32:12,219 INFO crawl.Generator - Generator: Selecting best-scoring urls due for fetch.
2011-02-16 14:32:12,220 INFO crawl.Generator - Generator: filtering: true
2011-02-16 14:32:12,220 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 14:32:12,224 INFO crawl.Generator - Generator: jobtracker is 'local', generating exactly one partition.
2011-02-16 14:32:14,409 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:14,782 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:14,783 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:14,784 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:14,785 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:14,809 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:14,810 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-16 14:32:14,810 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:15,003 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:15,223 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:15,224 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:15,225 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:15,226 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:15,240 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:15,241 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-16 14:32:15,241 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:16,040 INFO crawl.Generator - Generator: Partitioning selected urls for politeness.
2011-02-16 14:32:17,043 INFO crawl.Generator - Generator: segment: crawl/segments/20110216143217
2011-02-16 14:32:18,662 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:18,872 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:18,873 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:18,874 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:19,454 INFO crawl.Generator - Generator: finished at 2011-02-16 14:32:19, elapsed: 00:00:07
2011-02-16 14:32:20,955 INFO fetcher.Fetcher - Fetcher: starting at 2011-02-16 14:32:20
2011-02-16 14:32:20,956 INFO fetcher.Fetcher - Fetcher: segment: crawl/segments/20110216143217
2011-02-16 14:32:23,347 INFO fetcher.Fetcher - Fetcher: threads: 10
2011-02-16 14:32:23,374 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:23,374 INFO fetcher.Fetcher - QueueFeeder finished: total 2 records + hit by time limit :0
2011-02-16 14:32:23,741 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:23,742 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:23,743 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:23,744 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:23,782 INFO fetcher.Fetcher - fetching http://tv/
2011-02-16 14:32:23,782 INFO fetcher.Fetcher - fetching http://tfs/
2011-02-16 14:32:23,783 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,783 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,784 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,784 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=3
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,785 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=2
2011-02-16 14:32:23,839 INFO httpclient.Http - http.proxy.host = proxy.services.toyota.com<http://proxy.services.toyota.com>
2011-02-16 14:32:23,839 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 14:32:23,839 INFO httpclient.Http - http.timeout = 10000
2011-02-16 14:32:23,839 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 14:32:23,840 INFO httpclient.Http - http.agent = nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch 1.2; http://devcpd1.lexus.com/; carl.zha@theportgrp.com<ma...@theportgrp.com>)
2011-02-16 14:32:23,840 INFO httpclient.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 14:32:23,840 INFO httpclient.Http - protocol.plugin.check.blocking = false
2011-02-16 14:32:23,840 INFO httpclient.Http - protocol.plugin.check.robots = false
2011-02-16 14:32:23,853 INFO httpclient.Http - http.proxy.host = proxy.services.toyota.com<http://proxy.services.toyota.com>
2011-02-16 14:32:23,854 INFO httpclient.Http - http.proxy.port = 80
2011-02-16 14:32:23,854 INFO httpclient.Http - http.timeout = 10000
2011-02-16 14:32:23,854 INFO httpclient.Http - http.content.limit = 65536
2011-02-16 14:32:23,854 INFO httpclient.Http - http.agent = nutch-solr-integration-test/Nutch-1.2 (CPD Web Crawler using Nutch 1.2; http://devcpd1.lexus.com/; carl.zha@theportgrp.com<ma...@theportgrp.com>)
2011-02-16 14:32:23,854 INFO httpclient.Http - http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
2011-02-16 14:32:23,854 INFO httpclient.Http - protocol.plugin.check.blocking = false
2011-02-16 14:32:23,855 INFO httpclient.Http - protocol.plugin.check.robots = false
2011-02-16 14:32:24,016 DEBUG auth.AuthChallengeProcessor - Supported authentication schemes in the order of preference: [ntlm, digest, basic]
2011-02-16 14:32:24,017 INFO auth.AuthChallengeProcessor - ntlm authentication scheme selected
2011-02-16 14:32:24,017 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:24,017 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:24,111 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:24,111 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:24,791 INFO fetcher.Fetcher - -activeThreads=2, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:24,971 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:24,971 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:24,971 INFO httpclient.HttpMethodDirector - Failure authenticating with NTLM <any realm>@tfs:80
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor - Supported authentication schemes in the order of preference: [ntlm, digest, basic]
2011-02-16 14:32:25,054 INFO auth.AuthChallengeProcessor - ntlm authentication scheme selected
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:25,054 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:25,127 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:25,127 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:25,193 DEBUG auth.AuthChallengeProcessor - Using authentication scheme: ntlm
2011-02-16 14:32:25,193 DEBUG auth.AuthChallengeProcessor - Authorization challenge processed
2011-02-16 14:32:25,193 INFO httpclient.HttpMethodDirector - Failure authenticating with NTLM <any realm>@tfs:80
2011-02-16 14:32:25,497 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=1
2011-02-16 14:32:25,801 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:26,811 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:27,821 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:28,831 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:29,841 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:30,851 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:31,861 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:32,871 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:33,881 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:33,988 INFO api.RobotRulesParser - Couldn't get robots.txt for http://tv/: java.net.SocketTimeoutException: Read timed out
2011-02-16 14:32:34,891 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:35,901 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:36,910 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:37,921 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:38,931 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:39,940 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:40,950 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:41,960 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:42,970 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:43,980 INFO fetcher.Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:44,004 ERROR httpclient.Http - java.net.SocketTimeoutException: Read timed out
2011-02-16 14:32:44,005 ERROR httpclient.Http - at java.net.SocketInputStream.socketRead0(Native Method)
2011-02-16 14:32:44,005 ERROR httpclient.Http - at java.net.SocketInputStream.read(SocketInputStream.java:129)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
2011-02-16 14:32:44,006 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
2011-02-16 14:32:44,007 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
2011-02-16 14:32:44,008 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at org.apache.nutch.protocol.httpclient.HttpResponse.<init>(HttpResponse.java:94)
2011-02-16 14:32:44,010 ERROR httpclient.Http - at org.apache.nutch.protocol.httpclient.Http.getResponse(Http.java:154)
2011-02-16 14:32:44,011 ERROR httpclient.Http - at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:224)
2011-02-16 14:32:44,011 ERROR httpclient.Http - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:628)
2011-02-16 14:32:44,011 INFO fetcher.Fetcher - fetch of http://tv/ failed with: java.net.SocketTimeoutException: Read timed out
2011-02-16 14:32:44,012 INFO fetcher.Fetcher - -finishing thread FetcherThread, activeThreads=0
2011-02-16 14:32:44,990 INFO fetcher.Fetcher - -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
2011-02-16 14:32:44,990 INFO fetcher.Fetcher - -activeThreads=0
2011-02-16 14:32:45,234 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-02-16 14:32:46,236 INFO fetcher.Fetcher - Fetcher: finished at 2011-02-16 14:32:46, elapsed: 00:00:25
2011-02-16 14:32:47,763 INFO parse.ParseSegment - ParseSegment: starting at 2011-02-16 14:32:47
2011-02-16 14:32:47,764 INFO parse.ParseSegment - ParseSegment: segment: crawl/segments/20110216143217
2011-02-16 14:32:49,938 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-02-16 14:32:50,194 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:50,566 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:50,567 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:50,568 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:50,569 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:50,742 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:51,046 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:51,046 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:51,047 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:51,048 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:51,049 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:51,936 INFO parse.ParseSegment - ParseSegment: finished at 2011-02-16 14:32:51, elapsed: 00:00:04
2011-02-16 14:32:53,722 INFO crawl.CrawlDb - CrawlDb update: starting at 2011-02-16 14:32:53
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update: db: crawl/crawldb
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update: segments: [crawl/segments/20110216143217]
2011-02-16 14:32:53,723 INFO crawl.CrawlDb - CrawlDb update: additions allowed: true
2011-02-16 14:32:53,724 INFO crawl.CrawlDb - CrawlDb update: URL normalizing: true
2011-02-16 14:32:53,724 INFO crawl.CrawlDb - CrawlDb update: URL filtering: true
2011-02-16 14:32:53,753 INFO crawl.CrawlDb - CrawlDb update: Merging segment data into db.
2011-02-16 14:32:55,862 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:56,241 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:56,242 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:56,243 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:56,244 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:56,244 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:56,468 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-02-16 14:32:56,677 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:57,104 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:57,105 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,106 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,107 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,407 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:57,604 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:57,605 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,606 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,607 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,677 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:32:57,868 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:32:57,869 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:32:57,870 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:32:57,876 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:32:57,877 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-16 14:32:57,877 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:32:58,526 INFO crawl.CrawlDb - CrawlDb update: finished at 2011-02-16 14:32:58, elapsed: 00:00:04
2011-02-16 14:33:00,393 INFO crawl.Generator - Generator: starting at 2011-02-16 14:33:00
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: Selecting best-scoring urls due for fetch.
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: filtering: true
2011-02-16 14:33:00,394 INFO crawl.Generator - Generator: normalizing: true
2011-02-16 14:33:00,395 INFO crawl.Generator - Generator: topN: 1000
2011-02-16 14:33:00,398 INFO crawl.Generator - Generator: jobtracker is 'local', generating exactly one partition.
2011-02-16 14:33:02,568 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:33:02,949 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:33:02,950 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:33:02,951 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:33:02,952 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:33:02,975 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:33:02,976 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-16 14:33:02,977 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:33:03,168 INFO plugin.PluginRepository - Plugins: looking in: /www/nutch-1.2/plugins
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Plugin Auto-activation mode: [true]
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Registered Plugins:
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - the nutch core extension points (nutch-extensionpoints)
2011-02-16 14:33:03,380 INFO plugin.PluginRepository - Basic Query Filter (query-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Basic Indexing Filter (index-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Html Parse Plug-in (parse-html)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Basic Summarizer Plug-in (summary-basic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Http / Https Protocol Plug-in (protocol-httpclient)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Site Query Filter (query-site)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - HTTP Framework (lib-http)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Regex URL Filter (urlfilter-regex)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - XML Response Writer Plug-in (response-xml)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - OPIC Scoring Plug-in (scoring-opic)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - CyberNeko HTML Parser (lib-nekohtml)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Anchor Indexing Filter (index-anchor)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - URL Query Filter (query-url)
2011-02-16 14:33:03,381 INFO plugin.PluginRepository - Regex URL Filter Framework (lib-regex-filter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - JSON Response Writer Plug-in (response-json)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Registered Extension-Points:
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Segment Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-02-16 14:33:03,382 INFO plugin.PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2011-02-16 14:33:03,383 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2011-02-16 14:33:03,398 INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule
2011-02-16 14:33:03,398 INFO crawl.AbstractFetchSchedule - defaultInterval=2592000
2011-02-16 14:33:03,398 INFO crawl.AbstractFetchSchedule - maxInterval=7776000
2011-02-16 14:33:04,190 WARN crawl.Generator - Generator: 0 records selected for fetching, exiting ...
2011-02-16 14:33:06,042 INFO crawl.LinkDb - LinkDb: starting at 2011-02-16 14:33:06
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: URL normalize: true
2011-02-16 14:33:06,043 INFO crawl.LinkDb - LinkDb: URL filter: true
2011-02-16 14:33:06,061 INFO crawl.LinkDb - LinkDb: adding segment: file:/www/nutch-1.2/crawl/segments/20110216143217
2011-02-16 14:33:07,941 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-02-16 14:33:08,831 INFO crawl.LinkDb - LinkDb: finished at 2011-02-16 14:33:08, elapsed: 00:00:02