You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Chen, Richard" <ri...@intel.com> on 2014/02/20 07:09:36 UTC

hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/ hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split: hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)


Please kindly help me on this.

Thanks in advance!

Richard


RE: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by "Chen, Richard" <ri...@intel.com>.
Thanks for the reply!

I intended to run the job on a cluster of 5 physical nodes.
Please find attached my /etc/hadoop configuration directory (*.sh was renamed as *.sh_ to avoid mail fitering)

As you can see the configurations are pointing to samdev06 host where ResourceManager is deployed. I don’t know why it seems to run in a local mode.

I referenced to multiple sources for the configuration so I can’t give you one such source.

Richard

From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] On Behalf Of Vinod Kumar Vavilapalli
Sent: Friday, February 21, 2014 1:37 PM
To: user@hadoop.apache.org
Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com>> wrote:


Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)


Please kindly help me on this.

Thanks in advance!

Richard


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by "Chen, Richard" <ri...@intel.com>.
Thanks for the reply!

I intended to run the job on a cluster of 5 physical nodes.
Please find attached my /etc/hadoop configuration directory (*.sh was renamed as *.sh_ to avoid mail fitering)

As you can see the configurations are pointing to samdev06 host where ResourceManager is deployed. I don’t know why it seems to run in a local mode.

I referenced to multiple sources for the configuration so I can’t give you one such source.

Richard

From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] On Behalf Of Vinod Kumar Vavilapalli
Sent: Friday, February 21, 2014 1:37 PM
To: user@hadoop.apache.org
Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com>> wrote:


Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)


Please kindly help me on this.

Thanks in advance!

Richard


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by "Chen, Richard" <ri...@intel.com>.
Thanks for the reply!

I intended to run the job on a cluster of 5 physical nodes.
Please find attached my /etc/hadoop configuration directory (*.sh was renamed as *.sh_ to avoid mail fitering)

As you can see the configurations are pointing to samdev06 host where ResourceManager is deployed. I don’t know why it seems to run in a local mode.

I referenced to multiple sources for the configuration so I can’t give you one such source.

Richard

From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] On Behalf Of Vinod Kumar Vavilapalli
Sent: Friday, February 21, 2014 1:37 PM
To: user@hadoop.apache.org
Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com>> wrote:


Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)


Please kindly help me on this.

Thanks in advance!

Richard


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by "Chen, Richard" <ri...@intel.com>.
Thanks for the reply!

I intended to run the job on a cluster of 5 physical nodes.
Please find attached my /etc/hadoop configuration directory (*.sh was renamed as *.sh_ to avoid mail fitering)

As you can see the configurations are pointing to samdev06 host where ResourceManager is deployed. I don’t know why it seems to run in a local mode.

I referenced to multiple sources for the configuration so I can’t give you one such source.

Richard

From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] On Behalf Of Vinod Kumar Vavilapalli
Sent: Friday, February 21, 2014 1:37 PM
To: user@hadoop.apache.org
Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com>> wrote:


Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)


Please kindly help me on this.

Thanks in advance!

Richard


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com> wrote:

> Dear Hadoop User group,
>  
> This is my virgin post. So please bear with me if I made any mistakes at my early time here.
>  
> Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
> I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.
>  
> After careful configuration, when I tried running the 1st PI job example:
>  $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10
>  
> The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
> The exception message goes like this:
>  
> Starting Job
> 14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
> 14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
> 14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
> 14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
> 14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner:
> 14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
> 14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
> 14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
> 14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
> 14/02/20 13:34:13 WARN mapred.Task: Could not find output size
> java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
>                 at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
>                 at org.apache.hadoop.mapred.Task.done(Task.java:1048)
>                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
>                 at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
>                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:744)
>  
>  
> Please kindly help me on this.
>  
> Thanks in advance!
>  
> Richard


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com> wrote:

> Dear Hadoop User group,
>  
> This is my virgin post. So please bear with me if I made any mistakes at my early time here.
>  
> Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
> I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.
>  
> After careful configuration, when I tried running the 1st PI job example:
>  $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10
>  
> The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
> The exception message goes like this:
>  
> Starting Job
> 14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
> 14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
> 14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
> 14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
> 14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner:
> 14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
> 14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
> 14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
> 14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
> 14/02/20 13:34:13 WARN mapred.Task: Could not find output size
> java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
>                 at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
>                 at org.apache.hadoop.mapred.Task.done(Task.java:1048)
>                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
>                 at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
>                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:744)
>  
>  
> Please kindly help me on this.
>  
> Thanks in advance!
>  
> Richard


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com> wrote:

> Dear Hadoop User group,
>  
> This is my virgin post. So please bear with me if I made any mistakes at my early time here.
>  
> Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
> I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.
>  
> After careful configuration, when I tried running the 1st PI job example:
>  $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10
>  
> The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
> The exception message goes like this:
>  
> Starting Job
> 14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
> 14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
> 14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
> 14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
> 14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner:
> 14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
> 14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
> 14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
> 14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
> 14/02/20 13:34:13 WARN mapred.Task: Could not find output size
> java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
>                 at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
>                 at org.apache.hadoop.mapred.Task.done(Task.java:1048)
>                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
>                 at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
>                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:744)
>  
>  
> Please kindly help me on this.
>  
> Thanks in advance!
>  
> Richard


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Are you trying to run it on a cluster or locally? There is something called 'local mode' which is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better share the resource that you are following as instructions.

+Viond

On Feb 19, 2014, at 10:09 PM, Chen, Richard <ri...@intel.com> wrote:

> Dear Hadoop User group,
>  
> This is my virgin post. So please bear with me if I made any mistakes at my early time here.
>  
> Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
> I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit Cent OS 6.
>  
> After careful configuration, when I tried running the 1st PI job example:
>  $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10
>  
> The result was successfully calculated but some exception message was also dumped during the execution and I couldn’t figure out why.
> The exception message goes like this:
>  
> Starting Job
> 14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
> 14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
> 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
> 14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
> 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
> 14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
> 14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
> 14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 14/02/20 13:34:13 INFO mapred.LocalJobRunner:
> 14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
> 14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
> 14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
> 14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
> 14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
> 14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done. And is in the process of committing
> 14/02/20 13:34:13 WARN mapred.Task: Could not find output size
> java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
>                 at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
>                 at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
>                 at org.apache.hadoop.mapred.Task.done(Task.java:1048)
>                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
>                 at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
>                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                 at java.lang.Thread.run(Thread.java:744)
>  
>  
> Please kindly help me on this.
>  
> Thanks in advance!
>  
> Richard


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.