You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kishore Yellamraju <ki...@rocketfuelinc.com> on 2013/04/21 01:00:38 UTC
rack awareness in hadoop
All,
I have posted this question to CDH ML , but i guess i can post it here
because its a general hadoop question.
When the NN or JT gets the rack info, i guess it stores the info in memory.
can i ask you where in the JVM memory it will store the results ( perm gen
?) ? . I am getting "cannot allocate memory on NN and JT " and they have
more than enough memory. when i looked at JVM usage stats i can see it
doesnt have enough perm free space.so if its storing the values in perm gen
then there is a chance of this memory issues.
Thanks in advance !!!
exception that i see in logs :
java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
at org.apache.hadoop.util.Shell.run(Shell.java:188)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
at
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at
org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
at
org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
at
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
at
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 14 more
2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
initialization failed:
java.lang.NullPointerException
-Thanks
kishore kumar yellamraju |Ground control
operations|kishore@rocketfuel.com| 408.203.042
4
Re: rack awareness in hadoop
Posted by Mohit Anchlia <mo...@gmail.com>.
And don't forget to look at unlimit settings as well
Sent from my iPhone
On Apr 20, 2013, at 5:07 PM, Marcos Luis Ortiz Valmaseda <ma...@gmail.com> wrote:
> Like, Aaron say, this problem is related the Linux memory manager.
> You can tune it using the vm.overcommit_memory=1.
> Before to do any change, read all resources first:
> http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
> http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
> http://lwn.net/Articles/317814/
> http://www.tldp.org/LDP/tlk/mm/memory.html
>
> To learn more about how to tune kernel variables for Hadoop applications. Read these links too:
> First, the amazing Hadoop Operations´s book from Eric:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
>
> Hadoop Performance Tuning Guide from AMD:
> http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
>
> Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning Guide:
> http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
>
> Best wishes.
>
>
>
> 2013/4/20 Aaron Eng <ae...@maprtech.com>
>> The problem is probably not related to the JVM memory so much as the Linux memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148) which would imply this is happening when trying to create a new process. The initial malloc for the new process space is being denied by the memory manager. There could be many reasons why this happens, though the most likely is your overcommit settings and swap space. I'd suggest reading through these details:
>>
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>>
>> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <ki...@rocketfuelinc.com> wrote:
>>> All,
>>>
>>> I have posted this question to CDH ML , but i guess i can post it here because its a general hadoop question.
>>>
>>> When the NN or JT gets the rack info, i guess it stores the info in memory. can i ask you where in the JVM memory it will store the results ( perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and they have more than enough memory. when i looked at JVM usage stats i can see it doesnt have enough perm free space.so if its storing the values in perm gen then there is a chance of this memory issues.
>>>
>>>
>>> Thanks in advance !!!
>>>
>>>
>>> exception that i see in logs :
>>>
>>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>> at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>> at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>> at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>> at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>> at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ... 14 more
>>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:
>>> java.lang.NullPointerException
>>>
>>>
>>> -Thanks
>>> kishore kumar yellamraju |Ground control operations|kishore@rocketfuel.com | 408.203.0424
>>>
>>>
>
>
>
> --
> Marcos Ortiz Valmaseda,
> Data-Driven Product Manager at PDVSA
> Blog: http://dataddict.wordpress.com/
> LinkedIn: http://www.linkedin.com/in/marcosluis2186
> Twitter: @marcosluis2186
Re: rack awareness in hadoop
Posted by Mohit Anchlia <mo...@gmail.com>.
And don't forget to look at unlimit settings as well
Sent from my iPhone
On Apr 20, 2013, at 5:07 PM, Marcos Luis Ortiz Valmaseda <ma...@gmail.com> wrote:
> Like, Aaron say, this problem is related the Linux memory manager.
> You can tune it using the vm.overcommit_memory=1.
> Before to do any change, read all resources first:
> http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
> http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
> http://lwn.net/Articles/317814/
> http://www.tldp.org/LDP/tlk/mm/memory.html
>
> To learn more about how to tune kernel variables for Hadoop applications. Read these links too:
> First, the amazing Hadoop Operations´s book from Eric:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
>
> Hadoop Performance Tuning Guide from AMD:
> http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
>
> Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning Guide:
> http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
>
> Best wishes.
>
>
>
> 2013/4/20 Aaron Eng <ae...@maprtech.com>
>> The problem is probably not related to the JVM memory so much as the Linux memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148) which would imply this is happening when trying to create a new process. The initial malloc for the new process space is being denied by the memory manager. There could be many reasons why this happens, though the most likely is your overcommit settings and swap space. I'd suggest reading through these details:
>>
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>>
>> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <ki...@rocketfuelinc.com> wrote:
>>> All,
>>>
>>> I have posted this question to CDH ML , but i guess i can post it here because its a general hadoop question.
>>>
>>> When the NN or JT gets the rack info, i guess it stores the info in memory. can i ask you where in the JVM memory it will store the results ( perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and they have more than enough memory. when i looked at JVM usage stats i can see it doesnt have enough perm free space.so if its storing the values in perm gen then there is a chance of this memory issues.
>>>
>>>
>>> Thanks in advance !!!
>>>
>>>
>>> exception that i see in logs :
>>>
>>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>> at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>> at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>> at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>> at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>> at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ... 14 more
>>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:
>>> java.lang.NullPointerException
>>>
>>>
>>> -Thanks
>>> kishore kumar yellamraju |Ground control operations|kishore@rocketfuel.com | 408.203.0424
>>>
>>>
>
>
>
> --
> Marcos Ortiz Valmaseda,
> Data-Driven Product Manager at PDVSA
> Blog: http://dataddict.wordpress.com/
> LinkedIn: http://www.linkedin.com/in/marcosluis2186
> Twitter: @marcosluis2186
Re: rack awareness in hadoop
Posted by Mohit Anchlia <mo...@gmail.com>.
And don't forget to look at unlimit settings as well
Sent from my iPhone
On Apr 20, 2013, at 5:07 PM, Marcos Luis Ortiz Valmaseda <ma...@gmail.com> wrote:
> Like, Aaron say, this problem is related the Linux memory manager.
> You can tune it using the vm.overcommit_memory=1.
> Before to do any change, read all resources first:
> http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
> http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
> http://lwn.net/Articles/317814/
> http://www.tldp.org/LDP/tlk/mm/memory.html
>
> To learn more about how to tune kernel variables for Hadoop applications. Read these links too:
> First, the amazing Hadoop Operations´s book from Eric:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
>
> Hadoop Performance Tuning Guide from AMD:
> http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
>
> Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning Guide:
> http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
>
> Best wishes.
>
>
>
> 2013/4/20 Aaron Eng <ae...@maprtech.com>
>> The problem is probably not related to the JVM memory so much as the Linux memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148) which would imply this is happening when trying to create a new process. The initial malloc for the new process space is being denied by the memory manager. There could be many reasons why this happens, though the most likely is your overcommit settings and swap space. I'd suggest reading through these details:
>>
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>>
>> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <ki...@rocketfuelinc.com> wrote:
>>> All,
>>>
>>> I have posted this question to CDH ML , but i guess i can post it here because its a general hadoop question.
>>>
>>> When the NN or JT gets the rack info, i guess it stores the info in memory. can i ask you where in the JVM memory it will store the results ( perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and they have more than enough memory. when i looked at JVM usage stats i can see it doesnt have enough perm free space.so if its storing the values in perm gen then there is a chance of this memory issues.
>>>
>>>
>>> Thanks in advance !!!
>>>
>>>
>>> exception that i see in logs :
>>>
>>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>> at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>> at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>> at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>> at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>> at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ... 14 more
>>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:
>>> java.lang.NullPointerException
>>>
>>>
>>> -Thanks
>>> kishore kumar yellamraju |Ground control operations|kishore@rocketfuel.com | 408.203.0424
>>>
>>>
>
>
>
> --
> Marcos Ortiz Valmaseda,
> Data-Driven Product Manager at PDVSA
> Blog: http://dataddict.wordpress.com/
> LinkedIn: http://www.linkedin.com/in/marcosluis2186
> Twitter: @marcosluis2186
Re: rack awareness in hadoop
Posted by Mohit Anchlia <mo...@gmail.com>.
And don't forget to look at unlimit settings as well
Sent from my iPhone
On Apr 20, 2013, at 5:07 PM, Marcos Luis Ortiz Valmaseda <ma...@gmail.com> wrote:
> Like, Aaron say, this problem is related the Linux memory manager.
> You can tune it using the vm.overcommit_memory=1.
> Before to do any change, read all resources first:
> http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
> http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
> http://lwn.net/Articles/317814/
> http://www.tldp.org/LDP/tlk/mm/memory.html
>
> To learn more about how to tune kernel variables for Hadoop applications. Read these links too:
> First, the amazing Hadoop Operations´s book from Eric:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
>
> Hadoop Performance Tuning Guide from AMD:
> http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
>
> Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning Guide:
> http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
>
> Best wishes.
>
>
>
> 2013/4/20 Aaron Eng <ae...@maprtech.com>
>> The problem is probably not related to the JVM memory so much as the Linux memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148) which would imply this is happening when trying to create a new process. The initial malloc for the new process space is being denied by the memory manager. There could be many reasons why this happens, though the most likely is your overcommit settings and swap space. I'd suggest reading through these details:
>>
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>>
>> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <ki...@rocketfuelinc.com> wrote:
>>> All,
>>>
>>> I have posted this question to CDH ML , but i guess i can post it here because its a general hadoop question.
>>>
>>> When the NN or JT gets the rack info, i guess it stores the info in memory. can i ask you where in the JVM memory it will store the results ( perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and they have more than enough memory. when i looked at JVM usage stats i can see it doesnt have enough perm free space.so if its storing the values in perm gen then there is a chance of this memory issues.
>>>
>>>
>>> Thanks in advance !!!
>>>
>>>
>>> exception that i see in logs :
>>>
>>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>> at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>> at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>> at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>> at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>> at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>> at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ... 14 more
>>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:
>>> java.lang.NullPointerException
>>>
>>>
>>> -Thanks
>>> kishore kumar yellamraju |Ground control operations|kishore@rocketfuel.com | 408.203.0424
>>>
>>>
>
>
>
> --
> Marcos Ortiz Valmaseda,
> Data-Driven Product Manager at PDVSA
> Blog: http://dataddict.wordpress.com/
> LinkedIn: http://www.linkedin.com/in/marcosluis2186
> Twitter: @marcosluis2186
Re: rack awareness in hadoop
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Like, Aaron say, this problem is related the Linux memory manager.
You can tune it using the vm.overcommit_memory=1.
Before to do any change, read all resources first:
http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
http://lwn.net/Articles/317814/
http://www.tldp.org/LDP/tlk/mm/memory.html
To learn more about how to tune kernel variables for Hadoop applications.
Read these links too:
First, the amazing Hadoop Operations´s book from Eric:
http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
Hadoop Performance Tuning Guide from AMD:
http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning
Guide:
http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
Best wishes.
2013/4/20 Aaron Eng <ae...@maprtech.com>
> The problem is probably not related to the JVM memory so much as the Linux
> memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> which would imply this is happening when trying to create a new process.
> The initial malloc for the new process space is being denied by the memory
> manager. There could be many reasons why this happens, though the most
> likely is your overcommit settings and swap space. I'd suggest reading
> through these details:
>
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>
> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
> kishore@rocketfuelinc.com> wrote:
>
>> All,
>>
>> I have posted this question to CDH ML , but i guess i can post it here
>> because its a general hadoop question.
>>
>> When the NN or JT gets the rack info, i guess it stores the info in
>> memory. can i ask you where in the JVM memory it will store the results (
>> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
>> they have more than enough memory. when i looked at JVM usage stats i can
>> see it doesnt have enough perm free space.so if its storing the values in
>> perm gen then there is a chance of this memory issues.
>>
>>
>> Thanks in advance !!!
>>
>>
>> exception that i see in logs :
>>
>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
>> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
>> Cannot allocate memory
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>> at
>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>> at
>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>> at
>> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>> at
>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>> at
>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>> at
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>> ... 14 more
>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
>> initialization failed:
>> java.lang.NullPointerException
>>
>>
>> -Thanks
>> kishore kumar yellamraju |Ground control operations|
>> kishore@rocketfuel.com | 408.203.0424
>>
>>
>
--
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Blog*: http://dataddict.wordpress.com/
*LinkedIn: *http://www.linkedin.com/in/marcosluis2186
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
Re: rack awareness in hadoop
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Like, Aaron say, this problem is related the Linux memory manager.
You can tune it using the vm.overcommit_memory=1.
Before to do any change, read all resources first:
http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
http://lwn.net/Articles/317814/
http://www.tldp.org/LDP/tlk/mm/memory.html
To learn more about how to tune kernel variables for Hadoop applications.
Read these links too:
First, the amazing Hadoop Operations´s book from Eric:
http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
Hadoop Performance Tuning Guide from AMD:
http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning
Guide:
http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
Best wishes.
2013/4/20 Aaron Eng <ae...@maprtech.com>
> The problem is probably not related to the JVM memory so much as the Linux
> memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> which would imply this is happening when trying to create a new process.
> The initial malloc for the new process space is being denied by the memory
> manager. There could be many reasons why this happens, though the most
> likely is your overcommit settings and swap space. I'd suggest reading
> through these details:
>
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>
> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
> kishore@rocketfuelinc.com> wrote:
>
>> All,
>>
>> I have posted this question to CDH ML , but i guess i can post it here
>> because its a general hadoop question.
>>
>> When the NN or JT gets the rack info, i guess it stores the info in
>> memory. can i ask you where in the JVM memory it will store the results (
>> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
>> they have more than enough memory. when i looked at JVM usage stats i can
>> see it doesnt have enough perm free space.so if its storing the values in
>> perm gen then there is a chance of this memory issues.
>>
>>
>> Thanks in advance !!!
>>
>>
>> exception that i see in logs :
>>
>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
>> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
>> Cannot allocate memory
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>> at
>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>> at
>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>> at
>> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>> at
>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>> at
>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>> at
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>> ... 14 more
>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
>> initialization failed:
>> java.lang.NullPointerException
>>
>>
>> -Thanks
>> kishore kumar yellamraju |Ground control operations|
>> kishore@rocketfuel.com | 408.203.0424
>>
>>
>
--
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Blog*: http://dataddict.wordpress.com/
*LinkedIn: *http://www.linkedin.com/in/marcosluis2186
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
Re: rack awareness in hadoop
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Like, Aaron say, this problem is related the Linux memory manager.
You can tune it using the vm.overcommit_memory=1.
Before to do any change, read all resources first:
http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
http://lwn.net/Articles/317814/
http://www.tldp.org/LDP/tlk/mm/memory.html
To learn more about how to tune kernel variables for Hadoop applications.
Read these links too:
First, the amazing Hadoop Operations´s book from Eric:
http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
Hadoop Performance Tuning Guide from AMD:
http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning
Guide:
http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
Best wishes.
2013/4/20 Aaron Eng <ae...@maprtech.com>
> The problem is probably not related to the JVM memory so much as the Linux
> memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> which would imply this is happening when trying to create a new process.
> The initial malloc for the new process space is being denied by the memory
> manager. There could be many reasons why this happens, though the most
> likely is your overcommit settings and swap space. I'd suggest reading
> through these details:
>
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>
> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
> kishore@rocketfuelinc.com> wrote:
>
>> All,
>>
>> I have posted this question to CDH ML , but i guess i can post it here
>> because its a general hadoop question.
>>
>> When the NN or JT gets the rack info, i guess it stores the info in
>> memory. can i ask you where in the JVM memory it will store the results (
>> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
>> they have more than enough memory. when i looked at JVM usage stats i can
>> see it doesnt have enough perm free space.so if its storing the values in
>> perm gen then there is a chance of this memory issues.
>>
>>
>> Thanks in advance !!!
>>
>>
>> exception that i see in logs :
>>
>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
>> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
>> Cannot allocate memory
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>> at
>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>> at
>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>> at
>> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>> at
>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>> at
>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>> at
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>> ... 14 more
>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
>> initialization failed:
>> java.lang.NullPointerException
>>
>>
>> -Thanks
>> kishore kumar yellamraju |Ground control operations|
>> kishore@rocketfuel.com | 408.203.0424
>>
>>
>
--
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Blog*: http://dataddict.wordpress.com/
*LinkedIn: *http://www.linkedin.com/in/marcosluis2186
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
Re: rack awareness in hadoop
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Like, Aaron say, this problem is related the Linux memory manager.
You can tune it using the vm.overcommit_memory=1.
Before to do any change, read all resources first:
http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
http://lwn.net/Articles/317814/
http://www.tldp.org/LDP/tlk/mm/memory.html
To learn more about how to tune kernel variables for Hadoop applications.
Read these links too:
First, the amazing Hadoop Operations´s book from Eric:
http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
Hadoop Performance Tuning Guide from AMD:
http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
Intel® Distribution for Apache Hadoop* Software: Optimization and Tuning
Guide:
http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
Best wishes.
2013/4/20 Aaron Eng <ae...@maprtech.com>
> The problem is probably not related to the JVM memory so much as the Linux
> memory manager. The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> which would imply this is happening when trying to create a new process.
> The initial malloc for the new process space is being denied by the memory
> manager. There could be many reasons why this happens, though the most
> likely is your overcommit settings and swap space. I'd suggest reading
> through these details:
>
> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>
> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
> kishore@rocketfuelinc.com> wrote:
>
>> All,
>>
>> I have posted this question to CDH ML , but i guess i can post it here
>> because its a general hadoop question.
>>
>> When the NN or JT gets the rack info, i guess it stores the info in
>> memory. can i ask you where in the JVM memory it will store the results (
>> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
>> they have more than enough memory. when i looked at JVM usage stats i can
>> see it doesnt have enough perm free space.so if its storing the values in
>> perm gen then there is a chance of this memory issues.
>>
>>
>> Thanks in advance !!!
>>
>>
>> exception that i see in logs :
>>
>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
>> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
>> Cannot allocate memory
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>> at
>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>> at
>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>> at
>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>> at
>> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>> at
>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>> at
>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>> at
>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>> allocate memory
>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>> ... 14 more
>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
>> initialization failed:
>> java.lang.NullPointerException
>>
>>
>> -Thanks
>> kishore kumar yellamraju |Ground control operations|
>> kishore@rocketfuel.com | 408.203.0424
>>
>>
>
--
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Blog*: http://dataddict.wordpress.com/
*LinkedIn: *http://www.linkedin.com/in/marcosluis2186
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
Re: rack awareness in hadoop
Posted by Aaron Eng <ae...@maprtech.com>.
The problem is probably not related to the JVM memory so much as the Linux
memory manager. The exception is in
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
which would imply this is happening when trying to create a new process.
The initial malloc for the new process space is being denied by the memory
manager. There could be many reasons why this happens, though the most
likely is your overcommit settings and swap space. I'd suggest reading
through these details:
https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
kishore@rocketfuelinc.com> wrote:
> All,
>
> I have posted this question to CDH ML , but i guess i can post it here
> because its a general hadoop question.
>
> When the NN or JT gets the rack info, i guess it stores the info in
> memory. can i ask you where in the JVM memory it will store the results (
> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
> they have more than enough memory. when i looked at JVM usage stats i can
> see it doesnt have enough perm free space.so if its storing the values in
> perm gen then there is a chance of this memory issues.
>
>
> Thanks in advance !!!
>
>
> exception that i see in logs :
>
> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
> Cannot allocate memory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
> at
> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
> at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
> at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 14 more
> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.lang.NullPointerException
>
>
> -Thanks
> kishore kumar yellamraju |Ground control operations|
> kishore@rocketfuel.com | 408.203.0424
>
>
Re: rack awareness in hadoop
Posted by Aaron Eng <ae...@maprtech.com>.
The problem is probably not related to the JVM memory so much as the Linux
memory manager. The exception is in
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
which would imply this is happening when trying to create a new process.
The initial malloc for the new process space is being denied by the memory
manager. There could be many reasons why this happens, though the most
likely is your overcommit settings and swap space. I'd suggest reading
through these details:
https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
kishore@rocketfuelinc.com> wrote:
> All,
>
> I have posted this question to CDH ML , but i guess i can post it here
> because its a general hadoop question.
>
> When the NN or JT gets the rack info, i guess it stores the info in
> memory. can i ask you where in the JVM memory it will store the results (
> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
> they have more than enough memory. when i looked at JVM usage stats i can
> see it doesnt have enough perm free space.so if its storing the values in
> perm gen then there is a chance of this memory issues.
>
>
> Thanks in advance !!!
>
>
> exception that i see in logs :
>
> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
> Cannot allocate memory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
> at
> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
> at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
> at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 14 more
> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.lang.NullPointerException
>
>
> -Thanks
> kishore kumar yellamraju |Ground control operations|
> kishore@rocketfuel.com | 408.203.0424
>
>
Re: rack awareness in hadoop
Posted by Aaron Eng <ae...@maprtech.com>.
The problem is probably not related to the JVM memory so much as the Linux
memory manager. The exception is in
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
which would imply this is happening when trying to create a new process.
The initial malloc for the new process space is being denied by the memory
manager. There could be many reasons why this happens, though the most
likely is your overcommit settings and swap space. I'd suggest reading
through these details:
https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
kishore@rocketfuelinc.com> wrote:
> All,
>
> I have posted this question to CDH ML , but i guess i can post it here
> because its a general hadoop question.
>
> When the NN or JT gets the rack info, i guess it stores the info in
> memory. can i ask you where in the JVM memory it will store the results (
> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
> they have more than enough memory. when i looked at JVM usage stats i can
> see it doesnt have enough perm free space.so if its storing the values in
> perm gen then there is a chance of this memory issues.
>
>
> Thanks in advance !!!
>
>
> exception that i see in logs :
>
> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
> Cannot allocate memory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
> at
> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
> at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
> at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 14 more
> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.lang.NullPointerException
>
>
> -Thanks
> kishore kumar yellamraju |Ground control operations|
> kishore@rocketfuel.com | 408.203.0424
>
>
Re: rack awareness in hadoop
Posted by Aaron Eng <ae...@maprtech.com>.
The problem is probably not related to the JVM memory so much as the Linux
memory manager. The exception is in
java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
which would imply this is happening when trying to create a new process.
The initial malloc for the new process space is being denied by the memory
manager. There could be many reasons why this happens, though the most
likely is your overcommit settings and swap space. I'd suggest reading
through these details:
https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <
kishore@rocketfuelinc.com> wrote:
> All,
>
> I have posted this question to CDH ML , but i guess i can post it here
> because its a general hadoop question.
>
> When the NN or JT gets the rack info, i guess it stores the info in
> memory. can i ask you where in the JVM memory it will store the results (
> perm gen ?) ? . I am getting "cannot allocate memory on NN and JT " and
> they have more than enough memory. when i looked at JVM usage stats i can
> see it doesnt have enough perm free space.so if its storing the values in
> perm gen then there is a chance of this memory issues.
>
>
> Thanks in advance !!!
>
>
> exception that i see in logs :
>
> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in
> directory "/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12,
> Cannot allocate memory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
> at
> org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
> at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
> at
> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
> at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 14 more
> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.lang.NullPointerException
>
>
> -Thanks
> kishore kumar yellamraju |Ground control operations|
> kishore@rocketfuel.com | 408.203.0424
>
>