You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by mail list <lo...@gmail.com> on 2014/10/23 04:24:33 UTC
mapred job pending at "Starting scan to move intermediate done files"
hi, all,
I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:
louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
As you see, the job halt. Then I check the jps output:
louis@ubuntu:~/src/hadoop-2.5.1$ jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode
It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
Then i check the web ui:
It seems the job is pending!!!!
The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !
Re: mapred job pending at "Starting scan to move intermediate done files"
Posted by mail list <lo...@gmail.com>.
Hi，
Thanks Rohith，I just try this job on my virtual machine which does not have so much memory.
Can I modify some configuration options to let it run with fewer memory?
If not, i have to try it with a new environment.
Thank you again!
On Oct 23, 2014, at 14:18, Rohith Sharma K S <ro...@huawei.com> wrote:
> Hi,
>
> This is problem with your memory configurations in cluster. You have configured “yarn.nodemanager.resource.memory-mb” as 64MB which is too low.
>
> 1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
> 2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
>
> Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
>
> Thanks & Regards
> Rohith Sharma K S
>
> This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
>
> From: mail list [mailto:louis.hust.ml@gmail.com]
> Sent: 23 October 2014 07:55
> To: user@hadoop.apache.org
> Subject: mapred job pending at "Starting scan to move intermediate done files"
>
> hi, all,
>
> I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
> When I run a mapped job, the job output the following logs:
>
> louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
> 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
> 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
> 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job:http://localhost:8088/proxy/application_1414030015373_0001/
> 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
>
> As you see, the job halt. Then I check the jps output:
>
> louis@ubuntu:~/src/hadoop-2.5.1$ jps
> 22433 SecondaryNameNode
> 22716 NodeManager
> 22240 DataNode
> 22577 ResourceManager
> 23083 JobHistoryServer
> 23148 Jps
> 22080 NameNode
>
> It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
>
> 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
> 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
> 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
>
> Then i check the web ui:
> It seems the job is pending!!!!
>
> The attachment contains some configuration files at etc/hadoop/ .
> Any idea will be appreciated !
Re: mapred job pending at "Starting scan to move intermediate done files"
Posted by mail list <lo...@gmail.com>.
Hi，
Thanks Rohith，I just try this job on my virtual machine which does not have so much memory.
Can I modify some configuration options to let it run with fewer memory?
If not, i have to try it with a new environment.
Thank you again!
On Oct 23, 2014, at 14:18, Rohith Sharma K S <ro...@huawei.com> wrote:
> Hi,
>
> This is problem with your memory configurations in cluster. You have configured “yarn.nodemanager.resource.memory-mb” as 64MB which is too low.
>
> 1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
> 2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
>
> Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
>
> Thanks & Regards
> Rohith Sharma K S
>
> This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
>
> From: mail list [mailto:louis.hust.ml@gmail.com]
> Sent: 23 October 2014 07:55
> To: user@hadoop.apache.org
> Subject: mapred job pending at "Starting scan to move intermediate done files"
>
> hi, all,
>
> I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
> When I run a mapped job, the job output the following logs:
>
> louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
> 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
> 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
> 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job:http://localhost:8088/proxy/application_1414030015373_0001/
> 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
>
> As you see, the job halt. Then I check the jps output:
>
> louis@ubuntu:~/src/hadoop-2.5.1$ jps
> 22433 SecondaryNameNode
> 22716 NodeManager
> 22240 DataNode
> 22577 ResourceManager
> 23083 JobHistoryServer
> 23148 Jps
> 22080 NameNode
>
> It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
>
> 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
> 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
> 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
>
> Then i check the web ui:
> It seems the job is pending!!!!
>
> The attachment contains some configuration files at etc/hadoop/ .
> Any idea will be appreciated !
Re: mapred job pending at "Starting scan to move intermediate done files"
Posted by mail list <lo...@gmail.com>.
Hi，
Thanks Rohith，I just try this job on my virtual machine which does not have so much memory.
Can I modify some configuration options to let it run with fewer memory?
If not, i have to try it with a new environment.
Thank you again!
On Oct 23, 2014, at 14:18, Rohith Sharma K S <ro...@huawei.com> wrote:
> Hi,
>
> This is problem with your memory configurations in cluster. You have configured “yarn.nodemanager.resource.memory-mb” as 64MB which is too low.
>
> 1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
> 2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
>
> Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
>
> Thanks & Regards
> Rohith Sharma K S
>
> This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
>
> From: mail list [mailto:louis.hust.ml@gmail.com]
> Sent: 23 October 2014 07:55
> To: user@hadoop.apache.org
> Subject: mapred job pending at "Starting scan to move intermediate done files"
>
> hi, all,
>
> I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
> When I run a mapped job, the job output the following logs:
>
> louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
> 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
> 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
> 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job:http://localhost:8088/proxy/application_1414030015373_0001/
> 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
>
> As you see, the job halt. Then I check the jps output:
>
> louis@ubuntu:~/src/hadoop-2.5.1$ jps
> 22433 SecondaryNameNode
> 22716 NodeManager
> 22240 DataNode
> 22577 ResourceManager
> 23083 JobHistoryServer
> 23148 Jps
> 22080 NameNode
>
> It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
>
> 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
> 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
> 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
>
> Then i check the web ui:
> It seems the job is pending!!!!
>
> The attachment contains some configuration files at etc/hadoop/ .
> Any idea will be appreciated !
Re: mapred job pending at "Starting scan to move intermediate done files"
Posted by mail list <lo...@gmail.com>.
Hi，
Thanks Rohith，I just try this job on my virtual machine which does not have so much memory.
Can I modify some configuration options to let it run with fewer memory?
If not, i have to try it with a new environment.
Thank you again!
On Oct 23, 2014, at 14:18, Rohith Sharma K S <ro...@huawei.com> wrote:
> Hi,
>
> This is problem with your memory configurations in cluster. You have configured “yarn.nodemanager.resource.memory-mb” as 64MB which is too low.
>
> 1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
> 2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
>
> Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
>
> Thanks & Regards
> Rohith Sharma K S
>
> This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
>
> From: mail list [mailto:louis.hust.ml@gmail.com]
> Sent: 23 October 2014 07:55
> To: user@hadoop.apache.org
> Subject: mapred job pending at "Starting scan to move intermediate done files"
>
> hi, all,
>
> I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
> When I run a mapped job, the job output the following logs:
>
> louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
> 14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
> 14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
> 14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
> 14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job:http://localhost:8088/proxy/application_1414030015373_0001/
> 14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
>
> As you see, the job halt. Then I check the jps output:
>
> louis@ubuntu:~/src/hadoop-2.5.1$ jps
> 22433 SecondaryNameNode
> 22716 NodeManager
> 22240 DataNode
> 22577 ResourceManager
> 23083 JobHistoryServer
> 23148 Jps
> 22080 NameNode
>
> It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
>
> 2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
> 2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
> 2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
> 2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
>
> Then i check the web ui:
> It seems the job is pending!!!!
>
> The attachment contains some configuration files at etc/hadoop/ .
> Any idea will be appreciated !
RE: mapred job pending at "Starting scan to move intermediate done
files"
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi,
This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb" as 64MB which is too low.
1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
From: mail list [mailto:louis.hust.ml@gmail.com]
Sent: 23 October 2014 07:55
To: user@hadoop.apache.org
Subject: mapred job pending at "Starting scan to move intermediate done files"
hi, all,
I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:
louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
As you see, the job halt. Then I check the jps output:
louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode
It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
Then i check the web ui:
It seems the job is pending!!!!
The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !
RE: mapred job pending at "Starting scan to move intermediate done
files"
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi,
This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb" as 64MB which is too low.
1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
From: mail list [mailto:louis.hust.ml@gmail.com]
Sent: 23 October 2014 07:55
To: user@hadoop.apache.org
Subject: mapred job pending at "Starting scan to move intermediate done files"
hi, all,
I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:
louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
As you see, the job halt. Then I check the jps output:
louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode
It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
Then i check the web ui:
It seems the job is pending!!!!
The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !
RE: mapred job pending at "Starting scan to move intermediate done
files"
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi,
This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb" as 64MB which is too low.
1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
From: mail list [mailto:louis.hust.ml@gmail.com]
Sent: 23 October 2014 07:55
To: user@hadoop.apache.org
Subject: mapred job pending at "Starting scan to move intermediate done files"
hi, all,
I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:
louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
As you see, the job halt. Then I check the jps output:
louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode
It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
Then i check the web ui:
It seems the job is pending!!!!
The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !
RE: mapred job pending at "Starting scan to move intermediate done
files"
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi,
This is problem with your memory configurations in cluster. You have configured "yarn.nodemanager.resource.memory-mb" as 64MB which is too low.
1. ApplicationMaster required 2GB to launch container but cluster memory it self has 64MB. So container never get assigned.
2. Further steps, map memory is 64MB, but map.opts has 1024MB in mapred-site.xml. Again it is contradictory.
Change NodeManger memory to 8GB and map/reduce memory to 2GB. Try running job.
Thanks & Regards
Rohith Sharma K S
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
From: mail list [mailto:louis.hust.ml@gmail.com]
Sent: 23 October 2014 07:55
To: user@hadoop.apache.org
Subject: mapred job pending at "Starting scan to move intermediate done files"
hi, all,
I am new to hadoop, and I install the hadoop-2.5.1 on ubuntu with Pseudo-distributed mode.
When I run a mapped job, the job output the following logs:
louis@ubuntu:~/src/hadoop-book$ hadoop jar hadoop-examples.jar v3.MaxTemperatureDriver input/ncdc/all max-temp
14/10/22 19:09:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/10/22 19:09:57 INFO input.FileInputFormat: Total input paths to process : 2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: number of splits:2
14/10/22 19:09:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1414030015373_0001
14/10/22 19:09:58 INFO impl.YarnClientImpl: Submitted application application_1414030015373_0001
14/10/22 19:09:58 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1414030015373_0001/
14/10/22 19:09:58 INFO mapreduce.Job: Running job: job_1414030015373_0001
As you see, the job halt. Then I check the jps output:
louis@ubuntu:~/src/hadoop-2.5.1$<mailto:louis@ubuntu:~/src/hadoop-2.5.1$> jps
22433 SecondaryNameNode
22716 NodeManager
22240 DataNode
22577 ResourceManager
23083 JobHistoryServer
23148 Jps
22080 NameNode
It seems nothing wrong, then i check the mapred-louis-historyserver-ubuntu.log:
2014-10-22 19:09:03,831 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-10-22 19:09:03,837 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-10-22 19:11:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:14:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:17:33,832 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-10-22 19:20:33,830 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
Then i check the web ui:
It seems the job is pending!!!!
The attachment contains some configuration files at etc/hadoop/ .
Any idea will be appreciated !