You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Amin Bandeali <am...@oomz.com> on 2011/05/04 08:29:32 UTC

Error in Nutch Fetch

Running Nutch Crawler on a 3 slave hadoop cluster. On the third depth, the
reduces fail with the following exception:

2011-05-03 15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: fs.trash.interval; Ignoring. 2011-05-03
15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: mapred.local.dir; Ignoring. 2011-05-03
15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter:
mapred.tasktracker.reduce.tasks.maximum; Ignoring. 2011-05-03 15:36:00,787
WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.data.dir; Ignoring. 2011-05-03
15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: fs.checkpoint.dir; Ignoring. 2011-05-03
15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.permissions; Ignoring. 2011-05-03
15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.block.size; Ignoring. 2011-05-03
15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: mapred.tasktracker.map.tasks.maximum;
Ignoring. 2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter:
hadoop.rpc.socket.factory.class.default; Ignoring. 2011-05-03 15:36:00,788
WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.namenode.handler.count; Ignoring.
2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: mapred.child.ulimit; Ignoring.
2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: tasktracker.http.threads; Ignoring.
2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: mapred.job.tracker.handler.count;
Ignoring. 2011-05-03 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2011-05-03
15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.datanode.du.reserved; Ignoring.
2011-05-03 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.name.dir; Ignoring. 2011-05-03
15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
attempt to override final parameter: dfs.datanode.handler.count; Ignoring.
2011-05-03 15:36:00,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId= 2011-05-03
15:36:00,909 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager:
MemoryLimit=78643200, MaxSingleShuffleLimit=19660800 2011-05-03 15:36:00,916
INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201105022308_0101_r_000002_0 Thread started: Thread for merging
on-disk files 2011-05-03 15:36:00,916 INFO
org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
Thread waiting: Thread for merging on-disk files 2011-05-03 15:36:00,917
INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201105022308_0101_r_000002_0 Need another 2 map output(s) where 0 is
already in progress 2011-05-03 15:36:00,918 INFO
org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
Thread started: Thread for merging in memory files 2011-05-03 15:36:00,922
INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201105022308_0101_r_000002_0: Got 2 new map-outputs & number of
known map outputs is 2 2011-05-03 15:36:00,922 INFO
org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
Scheduled 2 of 2 known outputs (0 slow hosts and 0 dup hosts) 2011-05-03
15:36:00,939 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 5009514
bytes (5009514 raw bytes) into RAM from attempt_201105022308_0101_m_000001_0
2011-05-03 15:36:00,985 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
9412300 bytes (9412300 raw bytes) into RAM from
attempt_201105022308_0101_m_000000_0 2011-05-03 15:36:01,068 INFO
org.apache.hadoop.mapred.ReduceTask: Read 5009514 bytes from map-output for
attempt_201105022308_0101_m_000001_0 2011-05-03 15:36:01,070 INFO
org.apache.hadoop.mapred.ReduceTask: Rec #1 from
attempt_201105022308_0101_m_000001_0 -> (55, 116) from
ip-10-242-173-20.ec2.internal 2011-05-03 15:36:01,085 INFO
org.apache.hadoop.mapred.ReduceTask: Read 9412300 bytes from map-output for
attempt_201105022308_0101_m_000000_0 2011-05-03 15:36:01,086 INFO
org.apache.hadoop.mapred.ReduceTask: Rec #1 from
attempt_201105022308_0101_m_000000_0 -> (38, 40955) from
ip-10-84-65-166.ec2.internal 2011-05-03 15:36:01,921 INFO
org.apache.hadoop.mapred.ReduceTask: Closed ram manager 2011-05-03
15:36:01,923 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk
merge complete: 0 files left. 2011-05-03 15:36:01,979 INFO
org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 2
segments... 2011-05-03 15:36:01,982 INFO org.apache.hadoop.mapred.Merger:
Merging 2 sorted segments 2011-05-03 15:36:01,982 INFO
org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2
segments left of total size: 14421814 bytes 2011-05-03 15:36:02,080 INFO
org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
Merge of the 2 files in-memory complete. Local file is
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/output/map_1.out
of size 14421812 2011-05-03 15:36:02,081 INFO
org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 0 files left.
2011-05-03 15:36:02,081 INFO org.apache.hadoop.mapred.ReduceTask: Initiating
final on-disk merge with 1 files 2011-05-03 15:36:02,082 INFO
org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2011-05-03
15:36:02,090 INFO org.apache.hadoop.mapred.Merger: Down to the last
merge-pass, with 1 segments left of total size: 14421812 bytes 2011-05-03
15:36:02,126 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable 2011-05-03 15:36:02,127 INFO
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
15:36:02,135 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
compressor 2011-05-03 15:36:02,149 INFO
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
15:36:02,158 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
compressor 2011-05-03 15:36:02,166 INFO
org.apache.nutch.plugin.PluginRepository: Plugins: looking in:
/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/jars/plugins
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
Plugin Auto-activation mode: [true] 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Registered Plugins: 2011-05-03
15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: the nutch core
extension points (nutch-extensionpoints) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Basic Query Filter (query-basic)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Basic
URL Normalizer (urlnormalizer-basic) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Html Parse Plug-in (parse-html)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Basic
Indexing Filter (index-basic) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Basic Summarizer Plug-in
(summary-basic) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Site Query Filter (query-site)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: HTTP
Framework (lib-http) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: More Indexing Filter (index-more)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Text
Parse Plug-in (parse-text) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Regex URL Filter (urlfilter-regex)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
Pass-through URL Normalizer (urlnormalizer-pass) 2011-05-03 15:36:02,297
INFO org.apache.nutch.plugin.PluginRepository: More Query Filter
(query-more) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Http Protocol Plug-in
(protocol-http) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: XML Response Writer Plug-in
(response-xml) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Regex URL Normalizer
(urlnormalizer-regex) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: OPIC Scoring Plug-in
(scoring-opic) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Tika Parser Plug-in (parse-tika)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
CyberNeko HTML Parser (lib-nekohtml) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Anchor Indexing Filter
(index-anchor) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: JavaScript Parser (parse-js)
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: URL
Query Filter (query-url) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Regex URL Filter Framework
(lib-regex-filter) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: JSON Response Writer Plug-in
(response-json) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Registered Extension-Points:
2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Nutch
Summarizer (org.apache.nutch.searcher.Summarizer) 2011-05-03 15:36:02,297
INFO org.apache.nutch.plugin.PluginRepository: Nutch Protocol
(org.apache.nutch.protocol.Protocol) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Analysis
(org.apache.nutch.analysis.NutchAnalyzer) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Field Filter
(org.apache.nutch.indexer.field.FieldFilter) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: HTML Parse Filter
(org.apache.nutch.parse.HtmlParseFilter) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Query Filter
(org.apache.nutch.searcher.QueryFilter) 2011-05-03 15:36:02,297 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Search Results Response
Writer (org.apache.nutch.searcher.response.ResponseWriter) 2011-05-03
15:36:02,298 INFO org.apache.nutch.plugin.PluginRepository: Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer) 2011-05-03 15:36:02,298 INFO
org.apache.nutch.plugin.PluginRepository: Nutch URL Filter
(org.apache.nutch.net.URLFilter) 2011-05-03 15:36:02,298 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Online Search Results
Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer) 2011-05-03
15:36:02,298 INFO org.apache.nutch.plugin.PluginRepository: Nutch Indexing
Filter (org.apache.nutch.indexer.IndexingFilter) 2011-05-03 15:36:02,298
INFO org.apache.nutch.plugin.PluginRepository: Nutch Content Parser
(org.apache.nutch.parse.Parser) 2011-05-03 15:36:02,298 INFO
org.apache.nutch.plugin.PluginRepository: Nutch Scoring
(org.apache.nutch.scoring.ScoringFilter) 2011-05-03 15:36:02,298 INFO
org.apache.nutch.plugin.PluginRepository: Ontology Model Loader
(org.apache.nutch.ontology.Ontology) 2011-05-03 15:36:02,303 INFO
org.apache.hadoop.conf.Configuration: found resource crawl-urlfilter.txt at
file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/jars/crawl-urlfilter.txt
2011-05-03 15:36:02,342 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor 2011-05-03 15:36:02,358 INFO
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
15:36:02,386 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
compressor 2011-05-03 15:36:02,402 INFO
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
15:36:02,410 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
compressor 2011-05-03 15:36:02,436 WARN
org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer: can't find
rules for scope 'outlink', using default 2011-05-03 15:36:05,202 WARN
org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: java.lang.NullPointerException at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:163) at
org.apache.nutch.crawl.CrawlDatum.readFields(CrawlDatum.java:278) at
org.apache.nutch.util.GenericWritableConfigurable.readFields(GenericWritableConfigurable.java:54)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751)
at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691) at
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:227)
at
org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318) at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
Caused by: java.lang.NullPointerException at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
... 11 more
Any idea what would be causing this? Thanks in advance.
-- 
Amin Bandeali
Cell: 714.757.9544

Follow me on twitter
http://twitter.com/aminbandeali

DISCLAIMER
This e-mail is confidential and intended solely for the use of the
individual to whom it is addressed. If you have received this e-mail in
error please notify me. Although this message and any attachments are
believed to be free of any virus or other defect, it is the responsibility
of the recipient to ensure that it is virus free and no responsibility is
accepted by me for any loss or damage in any way arising from its use.

Re: Error in Nutch Fetch

Posted by Luis Cappa Banda <lu...@gmail.com>.
Hello.

Please, can you add your hadoop master-slave configuration? Try to re-crawl
and index with your current configuration and tell us if the error appears
again or not. Maybe it was a localizated and particular error due to non
connection reasons between master and slaves. Sometimes happen.

Re: Error in Nutch Fetch

Posted by Viksit Gaur <vi...@gmail.com>.
Amin,

Were you able to find a solution to this problem? I've run into the
exact same thing and was wondering if you had any solutions yet.

Cheers,
Viksit

On Tue, May 3, 2011 at 11:29 PM, Amin Bandeali <am...@oomz.com> wrote:
> Running Nutch Crawler on a 3 slave hadoop cluster. On the third depth, the
> reduces fail with the following exception:
>
> 2011-05-03 15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: fs.trash.interval; Ignoring. 2011-05-03
> 15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: mapred.local.dir; Ignoring. 2011-05-03
> 15:36:00,786 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter:
> mapred.tasktracker.reduce.tasks.maximum; Ignoring. 2011-05-03 15:36:00,787
> WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.data.dir; Ignoring. 2011-05-03
> 15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: fs.checkpoint.dir; Ignoring. 2011-05-03
> 15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.permissions; Ignoring. 2011-05-03
> 15:36:00,787 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.block.size; Ignoring. 2011-05-03
> 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: mapred.tasktracker.map.tasks.maximum;
> Ignoring. 2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter:
> hadoop.rpc.socket.factory.class.default; Ignoring. 2011-05-03 15:36:00,788
> WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.namenode.handler.count; Ignoring.
> 2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: mapred.child.ulimit; Ignoring.
> 2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: tasktracker.http.threads; Ignoring.
> 2011-05-03 15:36:00,788 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: mapred.job.tracker.handler.count;
> Ignoring. 2011-05-03 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: hadoop.tmp.dir; Ignoring. 2011-05-03
> 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.datanode.du.reserved; Ignoring.
> 2011-05-03 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.name.dir; Ignoring. 2011-05-03
> 15:36:00,789 WARN org.apache.hadoop.conf.Configuration:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/job.xml:a
> attempt to override final parameter: dfs.datanode.handler.count; Ignoring.
> 2011-05-03 15:36:00,808 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId= 2011-05-03
> 15:36:00,909 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager:
> MemoryLimit=78643200, MaxSingleShuffleLimit=19660800 2011-05-03 15:36:00,916
> INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201105022308_0101_r_000002_0 Thread started: Thread for merging
> on-disk files 2011-05-03 15:36:00,916 INFO
> org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
> Thread waiting: Thread for merging on-disk files 2011-05-03 15:36:00,917
> INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201105022308_0101_r_000002_0 Need another 2 map output(s) where 0 is
> already in progress 2011-05-03 15:36:00,918 INFO
> org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
> Thread started: Thread for merging in memory files 2011-05-03 15:36:00,922
> INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201105022308_0101_r_000002_0: Got 2 new map-outputs & number of
> known map outputs is 2 2011-05-03 15:36:00,922 INFO
> org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
> Scheduled 2 of 2 known outputs (0 slow hosts and 0 dup hosts) 2011-05-03
> 15:36:00,939 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 5009514
> bytes (5009514 raw bytes) into RAM from attempt_201105022308_0101_m_000001_0
> 2011-05-03 15:36:00,985 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 9412300 bytes (9412300 raw bytes) into RAM from
> attempt_201105022308_0101_m_000000_0 2011-05-03 15:36:01,068 INFO
> org.apache.hadoop.mapred.ReduceTask: Read 5009514 bytes from map-output for
> attempt_201105022308_0101_m_000001_0 2011-05-03 15:36:01,070 INFO
> org.apache.hadoop.mapred.ReduceTask: Rec #1 from
> attempt_201105022308_0101_m_000001_0 -> (55, 116) from
> ip-10-242-173-20.ec2.internal 2011-05-03 15:36:01,085 INFO
> org.apache.hadoop.mapred.ReduceTask: Read 9412300 bytes from map-output for
> attempt_201105022308_0101_m_000000_0 2011-05-03 15:36:01,086 INFO
> org.apache.hadoop.mapred.ReduceTask: Rec #1 from
> attempt_201105022308_0101_m_000000_0 -> (38, 40955) from
> ip-10-84-65-166.ec2.internal 2011-05-03 15:36:01,921 INFO
> org.apache.hadoop.mapred.ReduceTask: Closed ram manager 2011-05-03
> 15:36:01,923 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk
> merge complete: 0 files left. 2011-05-03 15:36:01,979 INFO
> org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 2
> segments... 2011-05-03 15:36:01,982 INFO org.apache.hadoop.mapred.Merger:
> Merging 2 sorted segments 2011-05-03 15:36:01,982 INFO
> org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2
> segments left of total size: 14421814 bytes 2011-05-03 15:36:02,080 INFO
> org.apache.hadoop.mapred.ReduceTask: attempt_201105022308_0101_r_000002_0
> Merge of the 2 files in-memory complete. Local file is
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/attempt_201105022308_0101_r_000002_0/output/map_1.out
> of size 14421812 2011-05-03 15:36:02,081 INFO
> org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 0 files left.
> 2011-05-03 15:36:02,081 INFO org.apache.hadoop.mapred.ReduceTask: Initiating
> final on-disk merge with 1 files 2011-05-03 15:36:02,082 INFO
> org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2011-05-03
> 15:36:02,090 INFO org.apache.hadoop.mapred.Merger: Down to the last
> merge-pass, with 1 segments left of total size: 14421812 bytes 2011-05-03
> 15:36:02,126 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable 2011-05-03 15:36:02,127 INFO
> org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
> 15:36:02,135 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
> compressor 2011-05-03 15:36:02,149 INFO
> org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
> 15:36:02,158 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
> compressor 2011-05-03 15:36:02,166 INFO
> org.apache.nutch.plugin.PluginRepository: Plugins: looking in:
> /mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/jars/plugins
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
> Plugin Auto-activation mode: [true] 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Registered Plugins: 2011-05-03
> 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: the nutch core
> extension points (nutch-extensionpoints) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Basic Query Filter (query-basic)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Basic
> URL Normalizer (urlnormalizer-basic) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Html Parse Plug-in (parse-html)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Basic
> Indexing Filter (index-basic) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Basic Summarizer Plug-in
> (summary-basic) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Site Query Filter (query-site)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: HTTP
> Framework (lib-http) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: More Indexing Filter (index-more)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Text
> Parse Plug-in (parse-text) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Regex URL Filter (urlfilter-regex)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
> Pass-through URL Normalizer (urlnormalizer-pass) 2011-05-03 15:36:02,297
> INFO org.apache.nutch.plugin.PluginRepository: More Query Filter
> (query-more) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Http Protocol Plug-in
> (protocol-http) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: XML Response Writer Plug-in
> (response-xml) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Regex URL Normalizer
> (urlnormalizer-regex) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: OPIC Scoring Plug-in
> (scoring-opic) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Tika Parser Plug-in (parse-tika)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository:
> CyberNeko HTML Parser (lib-nekohtml) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Anchor Indexing Filter
> (index-anchor) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: JavaScript Parser (parse-js)
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: URL
> Query Filter (query-url) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Regex URL Filter Framework
> (lib-regex-filter) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: JSON Response Writer Plug-in
> (response-json) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Registered Extension-Points:
> 2011-05-03 15:36:02,297 INFO org.apache.nutch.plugin.PluginRepository: Nutch
> Summarizer (org.apache.nutch.searcher.Summarizer) 2011-05-03 15:36:02,297
> INFO org.apache.nutch.plugin.PluginRepository: Nutch Protocol
> (org.apache.nutch.protocol.Protocol) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Analysis
> (org.apache.nutch.analysis.NutchAnalyzer) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Field Filter
> (org.apache.nutch.indexer.field.FieldFilter) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: HTML Parse Filter
> (org.apache.nutch.parse.HtmlParseFilter) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Query Filter
> (org.apache.nutch.searcher.QueryFilter) 2011-05-03 15:36:02,297 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Search Results Response
> Writer (org.apache.nutch.searcher.response.ResponseWriter) 2011-05-03
> 15:36:02,298 INFO org.apache.nutch.plugin.PluginRepository: Nutch URL
> Normalizer (org.apache.nutch.net.URLNormalizer) 2011-05-03 15:36:02,298 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch URL Filter
> (org.apache.nutch.net.URLFilter) 2011-05-03 15:36:02,298 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Online Search Results
> Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer) 2011-05-03
> 15:36:02,298 INFO org.apache.nutch.plugin.PluginRepository: Nutch Indexing
> Filter (org.apache.nutch.indexer.IndexingFilter) 2011-05-03 15:36:02,298
> INFO org.apache.nutch.plugin.PluginRepository: Nutch Content Parser
> (org.apache.nutch.parse.Parser) 2011-05-03 15:36:02,298 INFO
> org.apache.nutch.plugin.PluginRepository: Nutch Scoring
> (org.apache.nutch.scoring.ScoringFilter) 2011-05-03 15:36:02,298 INFO
> org.apache.nutch.plugin.PluginRepository: Ontology Model Loader
> (org.apache.nutch.ontology.Ontology) 2011-05-03 15:36:02,303 INFO
> org.apache.hadoop.conf.Configuration: found resource crawl-urlfilter.txt at
> file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_201105022308_0101/jars/crawl-urlfilter.txt
> 2011-05-03 15:36:02,342 INFO org.apache.hadoop.io.compress.CodecPool: Got
> brand-new compressor 2011-05-03 15:36:02,358 INFO
> org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
> 15:36:02,386 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
> compressor 2011-05-03 15:36:02,402 INFO
> org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor 2011-05-03
> 15:36:02,410 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new
> compressor 2011-05-03 15:36:02,436 WARN
> org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer: can't find
> rules for scope 'outlink', using default 2011-05-03 15:36:05,202 WARN
> org.apache.hadoop.mapred.TaskTracker: Error running child
> java.lang.RuntimeException: java.lang.NullPointerException at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
> at org.apache.hadoop.io.MapWritable.readFields(MapWritable.java:163) at
> org.apache.nutch.crawl.CrawlDatum.readFields(CrawlDatum.java:278) at
> org.apache.nutch.util.GenericWritableConfigurable.readFields(GenericWritableConfigurable.java:54)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751)
> at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691) at
> org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:227)
> at
> org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:39)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318) at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> Caused by: java.lang.NullPointerException at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
> ... 11 more
> Any idea what would be causing this? Thanks in advance.
> --
> Amin Bandeali
> Cell: 714.757.9544
>
> Follow me on twitter
> http://twitter.com/aminbandeali
>
> DISCLAIMER
> This e-mail is confidential and intended solely for the use of the
> individual to whom it is addressed. If you have received this e-mail in
> error please notify me. Although this message and any attachments are
> believed to be free of any virus or other defect, it is the responsibility
> of the recipient to ensure that it is virus free and no responsibility is
> accepted by me for any loss or damage in any way arising from its use.
>