You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bradford Stephens <br...@gmail.com> on 2010/09/18 10:47:43 UTC

Cannot run program "bash": java.io.IOException

Hey guys,

I'm running into issues when doing a moderate-size EMR job on 12 m1.large
nodes. Mappers and Reducers will randomly fail.

The EMR defaults are 2 mappers / 2 reducers per node. I've tried running
with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No
difference.

There are about 1,000 map tasks. Not very much data, maybe 50G at most?

My job fails to complete. Looking in syslog shows this:

java.io.IOException: Cannot run program "bash": java.io.IOException:
error=12, Cannot allocate memory
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
 at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
 at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
 at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238)
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
 at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
 ... 11 more



I would ask the EMR forums, but I think I may get faster feedback here :)


-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution.
Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social Media,
and Computer Science

Re: Cannot run program "bash": java.io.IOException

Posted by Bradford Stephens <br...@gmail.com>.
Awesome! I think that may have fixed it. Will have to see the final results
tomorrow...

On Sat, Sep 18, 2010 at 3:47 PM, Chris K Wensel <ch...@wensel.net> wrote:

> see if this helps matters
>
> --bootstrap-action
> s3://elasticmapreduce/bootstrap-actions/create-swap-file.rb --args
> "-E,/mnt/swap,1000"
>
> ckw
>
> On Sep 18, 2010, at 1:22 PM, Bradford Stephens wrote:
>
> > That could be a problem... this is Elastic MapReduce. Can I do that?
> Guess
> > I'll have to experiment.
> >
> > Cheers,
> > B
> >
> > On Sat, Sep 18, 2010 at 7:21 AM, Edward Capriolo <edlinuxguru@gmail.com
> >wrote:
> >
> >> This happens because child processes try to allocate the same memory
> >> as the parent. One way to solve this is setting memory overcommit to
> >> your linux system.
> >>
> >> On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens
> >> <br...@gmail.com> wrote:
> >>> Hey guys,
> >>>
> >>> I'm running into issues when doing a moderate-size EMR job on 12
> m1.large
> >>> nodes. Mappers and Reducers will randomly fail.
> >>>
> >>> The EMR defaults are 2 mappers / 2 reducers per node. I've tried
> running
> >>> with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No
> >>> difference.
> >>>
> >>> There are about 1,000 map tasks. Not very much data, maybe 50G at most?
> >>>
> >>> My job fails to complete. Looking in syslog shows this:
> >>>
> >>> java.io.IOException: Cannot run program "bash": java.io.IOException:
> >>> error=12, Cannot allocate memory
> >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
> >>> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> >>> at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
> >>> at
> >>>
> >>
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
> >>> at
> >>>
> >>
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
> >>> at
> >>>
> >>
> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
> >>> at
> >>>
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238)
> >>> at
> >>>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
> >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365)
> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
> >>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> >>> allocate memory
> >>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> >>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> >>> ... 11 more
> >>>
> >>>
> >>>
> >>> I would ask the EMR forums, but I think I may get faster feedback here
> :)
> >>>
> >>>
> >>> --
> >>> Bradford Stephens,
> >>> Founder, Drawn to Scale
> >>> drawntoscalehq.com
> >>> 727.697.7528
> >>>
> >>> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> >> solution.
> >>> Process, store, query, search, and serve all your data.
> >>>
> >>> http://www.roadtofailure.com -- The Fringes of Scalability, Social
> >> Media,
> >>> and Computer Science
> >>>
> >>
> >
> >
> >
> > --
> > Bradford Stephens,
> > Founder, Drawn to Scale
> > drawntoscalehq.com
> > 727.697.7528
> >
> > http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> solution.
> > Process, store, query, search, and serve all your data.
> >
> > http://www.roadtofailure.com -- The Fringes of Scalability, Social
> Media,
> > and Computer Science
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://www.concurrentinc.com
>
> -- Concurrent, Inc. offers mentoring, support, and licensing for Cascading
>
>


-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution.
Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social Media,
and Computer Science

Re: Cannot run program "bash": java.io.IOException

Posted by Chris K Wensel <ch...@wensel.net>.
see if this helps matters

--bootstrap-action s3://elasticmapreduce/bootstrap-actions/create-swap-file.rb --args "-E,/mnt/swap,1000"

ckw

On Sep 18, 2010, at 1:22 PM, Bradford Stephens wrote:

> That could be a problem... this is Elastic MapReduce. Can I do that? Guess
> I'll have to experiment.
> 
> Cheers,
> B
> 
> On Sat, Sep 18, 2010 at 7:21 AM, Edward Capriolo <ed...@gmail.com>wrote:
> 
>> This happens because child processes try to allocate the same memory
>> as the parent. One way to solve this is setting memory overcommit to
>> your linux system.
>> 
>> On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens
>> <br...@gmail.com> wrote:
>>> Hey guys,
>>> 
>>> I'm running into issues when doing a moderate-size EMR job on 12 m1.large
>>> nodes. Mappers and Reducers will randomly fail.
>>> 
>>> The EMR defaults are 2 mappers / 2 reducers per node. I've tried running
>>> with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No
>>> difference.
>>> 
>>> There are about 1,000 map tasks. Not very much data, maybe 50G at most?
>>> 
>>> My job fails to complete. Looking in syslog shows this:
>>> 
>>> java.io.IOException: Cannot run program "bash": java.io.IOException:
>>> error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:134)
>>> at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
>>> at
>>> 
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
>>> at
>>> 
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>>> at
>>> 
>> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
>>> at
>>> 
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238)
>>> at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
>>> allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>>> ... 11 more
>>> 
>>> 
>>> 
>>> I would ask the EMR forums, but I think I may get faster feedback here :)
>>> 
>>> 
>>> --
>>> Bradford Stephens,
>>> Founder, Drawn to Scale
>>> drawntoscalehq.com
>>> 727.697.7528
>>> 
>>> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
>> solution.
>>> Process, store, query, search, and serve all your data.
>>> 
>>> http://www.roadtofailure.com -- The Fringes of Scalability, Social
>> Media,
>>> and Computer Science
>>> 
>> 
> 
> 
> 
> -- 
> Bradford Stephens,
> Founder, Drawn to Scale
> drawntoscalehq.com
> 727.697.7528
> 
> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution.
> Process, store, query, search, and serve all your data.
> 
> http://www.roadtofailure.com -- The Fringes of Scalability, Social Media,
> and Computer Science

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading

Re: Cannot run program "bash": java.io.IOException

Posted by Bradford Stephens <br...@gmail.com>.
That could be a problem... this is Elastic MapReduce. Can I do that? Guess
I'll have to experiment.

Cheers,
B

On Sat, Sep 18, 2010 at 7:21 AM, Edward Capriolo <ed...@gmail.com>wrote:

> This happens because child processes try to allocate the same memory
> as the parent. One way to solve this is setting memory overcommit to
> your linux system.
>
> On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens
> <br...@gmail.com> wrote:
> > Hey guys,
> >
> > I'm running into issues when doing a moderate-size EMR job on 12 m1.large
> > nodes. Mappers and Reducers will randomly fail.
> >
> > The EMR defaults are 2 mappers / 2 reducers per node. I've tried running
> > with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No
> > difference.
> >
> > There are about 1,000 map tasks. Not very much data, maybe 50G at most?
> >
> > My job fails to complete. Looking in syslog shows this:
> >
> > java.io.IOException: Cannot run program "bash": java.io.IOException:
> > error=12, Cannot allocate memory
> >  at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
> >  at org.apache.hadoop.util.Shell.run(Shell.java:134)
> > at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
> >  at
> >
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
> > at
> >
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
> >  at
> >
> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
> > at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238)
> >  at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> > allocate memory
> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> >  at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> >  ... 11 more
> >
> >
> >
> > I would ask the EMR forums, but I think I may get faster feedback here :)
> >
> >
> > --
> > Bradford Stephens,
> > Founder, Drawn to Scale
> > drawntoscalehq.com
> > 727.697.7528
> >
> > http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> solution.
> > Process, store, query, search, and serve all your data.
> >
> > http://www.roadtofailure.com -- The Fringes of Scalability, Social
> Media,
> > and Computer Science
> >
>



-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution.
Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social Media,
and Computer Science

Re: Cannot run program "bash": java.io.IOException

Posted by Edward Capriolo <ed...@gmail.com>.
This happens because child processes try to allocate the same memory
as the parent. One way to solve this is setting memory overcommit to
your linux system.

On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens
<br...@gmail.com> wrote:
> Hey guys,
>
> I'm running into issues when doing a moderate-size EMR job on 12 m1.large
> nodes. Mappers and Reducers will randomly fail.
>
> The EMR defaults are 2 mappers / 2 reducers per node. I've tried running
> with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No
> difference.
>
> There are about 1,000 map tasks. Not very much data, maybe 50G at most?
>
> My job fails to complete. Looking in syslog shows this:
>
> java.io.IOException: Cannot run program "bash": java.io.IOException:
> error=12, Cannot allocate memory
>  at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
>  at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
>  at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>  at
> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>  at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>  ... 11 more
>
>
>
> I would ask the EMR forums, but I think I may get faster feedback here :)
>
>
> --
> Bradford Stephens,
> Founder, Drawn to Scale
> drawntoscalehq.com
> 727.697.7528
>
> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data solution.
> Process, store, query, search, and serve all your data.
>
> http://www.roadtofailure.com -- The Fringes of Scalability, Social Media,
> and Computer Science
>