You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Andre Kelpe <ak...@concurrentinc.com> on 2013/08/29 16:52:52 UTC
[yarn] job is not getting assigned
Hi,
I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop
2.1-beta tarball.
My cluster has 1 master node running the hdfs namenode, the resourcemanger
and the job history server. Next to that I have 3 nodes acting as
datanodes and nodemanagers.
In order to test, if everything is working, I submitted the teragen job
from the hadoop-examples jar like this:
$ hadoop jar
$HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar
teragen 1000 /user/vagrant/teragen
The job starts up and I get the following output:
13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at
master.local/192.168.7.10:8032
13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated.
Instead, use mapreduce.job.user.name
13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated.
Instead, use mapreduce.job.jar
13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is
deprecated. Instead, use mapreduce.job.reduces
13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated.
Instead, use mapreduce.job.name
13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is
deprecated. Instead, use mapreduce.job.inputformat.class
13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is
deprecated. Instead, use mapreduce.job.outputformat.class
13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is
deprecated. Instead, use mapreduce.job.working.dir
13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1377787324271_0001
13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application
application_1377787324271_0001 to ResourceManager at master.local/
192.168.7.10:8032
13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job:
http://master.local:8088/proxy/application_1377787324271_0001/
13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
and then it stops. If I check the UI, I see this:
application_1377787324271_0001<http://master.local:8088/cluster/app/application_1377787324271_0001>
vagrantTeraGenMAPREDUCEdefaultThu, 29 Aug 2013 14:42:49 GMTN/AACCEPTED
UNDEFINED
UNASSIGNED <http://master.local:8088/cluster/apps#>
I have no idea, why it is not starting, nor what to look for. Any pointers
are more than welcome!
Thanks!
- André
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: [yarn] job is not getting assigned
Posted by Andre Kelpe <ak...@concurrentinc.com>.
Hi Vinod,
I found the issue: The yarn.nodemanager.resource.memory-mb value was to
low. I set it back to the default value and the job runs fine now.
Thanks!
- André
On Thu, Aug 29, 2013 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
>
> This usually means there are no available resources as seen by the
> ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If
> not, you'll have to check the NodeManager logs to see if they crashed for
> some reason.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
>
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop
> 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger
> and the job history server. Next to that I have 3 nodes acting as
> datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job
> from the hadoop-examples jar like this:
>
> $ hadoop jar
> $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar
> teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at
> master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application
> application_1377787324271_0001 to ResourceManager at master.local/
> 192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job:
> http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001<http://master.local:8088/cluster/app/application_1377787324271_0001>
> vagrantTeraGenMAPREDUCEdefaultThu, 29 Aug 2013 14:42:49 GMTN/AACCEPTED
> UNDEFINED
>
> UNASSIGNED <http://master.local:8088/cluster/apps#>
> I have no idea, why it is not starting, nor what to look for. Any pointers
> are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: [yarn] job is not getting assigned
Posted by Andre Kelpe <ak...@concurrentinc.com>.
Hi Vinod,
I found the issue: The yarn.nodemanager.resource.memory-mb value was to
low. I set it back to the default value and the job runs fine now.
Thanks!
- André
On Thu, Aug 29, 2013 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
>
> This usually means there are no available resources as seen by the
> ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If
> not, you'll have to check the NodeManager logs to see if they crashed for
> some reason.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
>
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop
> 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger
> and the job history server. Next to that I have 3 nodes acting as
> datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job
> from the hadoop-examples jar like this:
>
> $ hadoop jar
> $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar
> teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at
> master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application
> application_1377787324271_0001 to ResourceManager at master.local/
> 192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job:
> http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001<http://master.local:8088/cluster/app/application_1377787324271_0001>
> vagrantTeraGenMAPREDUCEdefaultThu, 29 Aug 2013 14:42:49 GMTN/AACCEPTED
> UNDEFINED
>
> UNASSIGNED <http://master.local:8088/cluster/apps#>
> I have no idea, why it is not starting, nor what to look for. Any pointers
> are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: [yarn] job is not getting assigned
Posted by Andre Kelpe <ak...@concurrentinc.com>.
Hi Vinod,
I found the issue: The yarn.nodemanager.resource.memory-mb value was to
low. I set it back to the default value and the job runs fine now.
Thanks!
- André
On Thu, Aug 29, 2013 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
>
> This usually means there are no available resources as seen by the
> ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If
> not, you'll have to check the NodeManager logs to see if they crashed for
> some reason.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
>
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop
> 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger
> and the job history server. Next to that I have 3 nodes acting as
> datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job
> from the hadoop-examples jar like this:
>
> $ hadoop jar
> $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar
> teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at
> master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application
> application_1377787324271_0001 to ResourceManager at master.local/
> 192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job:
> http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001<http://master.local:8088/cluster/app/application_1377787324271_0001>
> vagrantTeraGenMAPREDUCEdefaultThu, 29 Aug 2013 14:42:49 GMTN/AACCEPTED
> UNDEFINED
>
> UNASSIGNED <http://master.local:8088/cluster/apps#>
> I have no idea, why it is not starting, nor what to look for. Any pointers
> are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: [yarn] job is not getting assigned
Posted by Andre Kelpe <ak...@concurrentinc.com>.
Hi Vinod,
I found the issue: The yarn.nodemanager.resource.memory-mb value was to
low. I set it back to the default value and the job runs fine now.
Thanks!
- André
On Thu, Aug 29, 2013 at 7:36 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:
>
> This usually means there are no available resources as seen by the
> ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If
> not, you'll have to check the NodeManager logs to see if they crashed for
> some reason.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
>
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop
> 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger
> and the job history server. Next to that I have 3 nodes acting as
> datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job
> from the hadoop-examples jar like this:
>
> $ hadoop jar
> $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar
> teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at
> master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application
> application_1377787324271_0001 to ResourceManager at master.local/
> 192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job:
> http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001<http://master.local:8088/cluster/app/application_1377787324271_0001>
> vagrantTeraGenMAPREDUCEdefaultThu, 29 Aug 2013 14:42:49 GMTN/AACCEPTED
> UNDEFINED
>
> UNASSIGNED <http://master.local:8088/cluster/apps#>
> I have no idea, why it is not starting, nor what to look for. Any pointers
> are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: [yarn] job is not getting assigned
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This usually means there are no available resources as seen by the ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If not, you'll have to check the NodeManager logs to see if they crashed for some reason.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger and the job history server. Next to that I have 3 nodes acting as datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job from the hadoop-examples jar like this:
>
> $ hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application application_1377787324271_0001 to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job: http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001 vagrant TeraGen MAPREDUCE default Thu, 29 Aug 2013 14:42:49 GMT N/A ACCEPTED UNDEFINED
>
> UNASSIGNED
>
> I have no idea, why it is not starting, nor what to look for. Any pointers are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [yarn] job is not getting assigned
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This usually means there are no available resources as seen by the ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If not, you'll have to check the NodeManager logs to see if they crashed for some reason.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger and the job history server. Next to that I have 3 nodes acting as datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job from the hadoop-examples jar like this:
>
> $ hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application application_1377787324271_0001 to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job: http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001 vagrant TeraGen MAPREDUCE default Thu, 29 Aug 2013 14:42:49 GMT N/A ACCEPTED UNDEFINED
>
> UNASSIGNED
>
> I have no idea, why it is not starting, nor what to look for. Any pointers are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [yarn] job is not getting assigned
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This usually means there are no available resources as seen by the ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If not, you'll have to check the NodeManager logs to see if they crashed for some reason.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger and the job history server. Next to that I have 3 nodes acting as datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job from the hadoop-examples jar like this:
>
> $ hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application application_1377787324271_0001 to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job: http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001 vagrant TeraGen MAPREDUCE default Thu, 29 Aug 2013 14:42:49 GMT N/A ACCEPTED UNDEFINED
>
> UNASSIGNED
>
> I have no idea, why it is not starting, nor what to look for. Any pointers are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [yarn] job is not getting assigned
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
This usually means there are no available resources as seen by the ResourceManager. Do you see "Active Nodes" on the RM web UI first page? If not, you'll have to check the NodeManager logs to see if they crashed for some reason.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Aug 29, 2013, at 7:52 AM, Andre Kelpe wrote:
> Hi,
>
> I am in the middle of setting up a hadoop 2 cluster. I am using the hadoop 2.1-beta tarball.
>
> My cluster has 1 master node running the hdfs namenode, the resourcemanger and the job history server. Next to that I have 3 nodes acting as datanodes and nodemanagers.
>
> In order to test, if everything is working, I submitted the teragen job from the hadoop-examples jar like this:
>
> $ hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.1.0-beta.jar teragen 1000 /user/vagrant/teragen
>
> The job starts up and I get the following output:
>
> 13/08/29 14:42:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 13/08/29 14:42:47 INFO client.RMProxy: Connecting to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:48 INFO terasort.TeraSort: Generating 1000 using 2
> 13/08/29 14:42:48 INFO mapreduce.JobSubmitter: number of splits:2
> 13/08/29 14:42:48 WARN conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/29 14:42:48 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/29 14:42:48 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
> 13/08/29 14:42:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1377787324271_0001
> 13/08/29 14:42:50 INFO impl.YarnClientImpl: Submitted application application_1377787324271_0001 to ResourceManager at master.local/192.168.7.10:8032
> 13/08/29 14:42:50 INFO mapreduce.Job: The url to track the job: http://master.local:8088/proxy/application_1377787324271_0001/
> 13/08/29 14:42:50 INFO mapreduce.Job: Running job: job_1377787324271_0001
>
> and then it stops. If I check the UI, I see this:
>
> application_1377787324271_0001 vagrant TeraGen MAPREDUCE default Thu, 29 Aug 2013 14:42:49 GMT N/A ACCEPTED UNDEFINED
>
> UNASSIGNED
>
> I have no idea, why it is not starting, nor what to look for. Any pointers are more than welcome!
>
> Thanks!
>
> - André
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.