You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Keith Wiley <kw...@keithwiley.com> on 2012/08/01 22:33:47 UTC

Exec hadoop from Java, reuse JVM (client-side)?

I know there is a lot of discussion about JVM reuse in Hadoop, but that usually refers to mappers running on the cluste nodesr.  I have a much different question.  I am running a Java program which at one point execs hadoop and that call sometimes fails in the fashion shown below.  Thus, this issue occurs entirely within the client machine (of course, I am currently running in pseudo-distributed mode which convolutes that point somewhat).  In other words, I successfully ran a Java program, but it failed to subsequently run *another* Java program (hadoop).  My interpretation of the hadoop startup scripts (the hadoop command itself for example) is that they run a second JVM in my scenario, and that they fail to allocate enough memory.

Is there any way to run hadoop from within a JVM such that it reuses the local JVM?

EXCEPTION: java.io.IOException: Cannot run program "hadoop": java.io.IOException: error=12, Cannot allocate memory
    java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    java.lang.Runtime.exec(Runtime.java:593)
    java.lang.Runtime.exec(Runtime.java:466)
    com.util.Shell.run(Shell.java:44)
    com.exe.Foo.bar(Foo.java:107)
    com.exe.Foo.run(Foo.java:205)
    com.exe.Foo.main(Foo.java:227)
Exception in thread "main" java.io.IOException: Cannot run program "hadoop": java.io.IOException: error=12, Cannot allocate memory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
	at java.lang.Runtime.exec(Runtime.java:593)
	at java.lang.Runtime.exec(Runtime.java:466)
	at com.util.Shell.run(Shell.java:44)
	at com.exe.Foo.bar(Foo.java:107)
	at com.exe.Foo.run(Foo.java:205)
	at com.exe.Foo.main(Foo.java:227)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
	at java.lang.ProcessImpl.start(ProcessImpl.java:65)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
	... 6 more

________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"You can scratch an itch, but you can't itch a scratch. Furthermore, an itch can
itch but a scratch can't scratch. Finally, a scratch can itch, but an itch can't
scratch. All together this implies: He scratched the itch from the scratch that
itched but would never itch the scratch from the itch that scratched."
                                           --  Keith Wiley
________________________________________________________________________________

Re: Exec hadoop from Java, reuse JVM (client-side)?

Posted by Keith Wiley <kw...@keithwiley.com>.

It's just easier that way.  I don't have to link in any hadoop libraries or bring in any other hadoop related code.  It keeps the two environments fundamentally separated.  I suppose I could wrap hadoop into the exterior code, but I do kinda like the idea of keeping my various worlds separate.  I'll consider it, but I don't really like the idea.  I don't want the program to be very dependent on hadoop.  Simply removing a call to execing it is a lot easier than gutting hadoop code and linked .jars.

I'll take a look at it, maybe there's a way to do that with relative ease.

On Aug 1, 2012, at 17:57 , Jim Donofrio wrote:

> Why would you call the hadoop script, why not just call the part of the hadoop shell api you are trying to call directly from java?
> 
> 
> On 08/01/2012 07:37 PM, Keith Wiley wrote:
>> Hmmm, at first glance that does appear to be similar to my situation.  I'll have to delve through it in detail to see if it squarely addresses (and fixes) my problem.  Mine is sporadic and I suspect dependent on the current memory situation (it isn't a deterministic and guaranteed failure).  I am not sure if that is true of the stackoverflow question you referenced...but it is certainly worth reading over.
>> 
>> Thanks.
>> 
>> On Aug 1, 2012, at 15:34 , Dhruv wrote:
>> 
>>> Is this related?
>>> 
>>> http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run

________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"Luminous beings are we, not this crude matter."
                                           --  Yoda
________________________________________________________________________________

Re: Exec hadoop from Java, reuse JVM (client-side)?

Posted by Jim Donofrio <do...@gmail.com>.

Why would you call the hadoop script, why not just call the part of the 
hadoop shell api you are trying to call directly from java?


On 08/01/2012 07:37 PM, Keith Wiley wrote:
> Hmmm, at first glance that does appear to be similar to my situation.  I'll have to delve through it in detail to see if it squarely addresses (and fixes) my problem.  Mine is sporadic and I suspect dependent on the current memory situation (it isn't a deterministic and guaranteed failure).  I am not sure if that is true of the stackoverflow question you referenced...but it is certainly worth reading over.
>
> Thanks.
>
> On Aug 1, 2012, at 15:34 , Dhruv wrote:
>
>> Is this related?
>>
>> http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run
>
> ________________________________________________________________________________
> Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com
>
> "It's a fine line between meticulous and obsessive-compulsive and a slippery
> rope between obsessive-compulsive and debilitatingly slow."
>                                             --  Keith Wiley
> ________________________________________________________________________________
>
>

Re: Exec hadoop from Java, reuse JVM (client-side)?

Posted by Keith Wiley <kw...@keithwiley.com>.

Hmmm, at first glance that does appear to be similar to my situation.  I'll have to delve through it in detail to see if it squarely addresses (and fixes) my problem.  Mine is sporadic and I suspect dependent on the current memory situation (it isn't a deterministic and guaranteed failure).  I am not sure if that is true of the stackoverflow question you referenced...but it is certainly worth reading over.

Thanks.

On Aug 1, 2012, at 15:34 , Dhruv wrote:

> Is this related?
> 
> http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run

________________________________________________________________________________
Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"It's a fine line between meticulous and obsessive-compulsive and a slippery
rope between obsessive-compulsive and debilitatingly slow."
                                           --  Keith Wiley
________________________________________________________________________________

Re: Exec hadoop from Java, reuse JVM (client-side)?

Posted by Dhruv <dh...@gmail.com>.

Is this related?

http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error-12-cannot-allocate-memory-calling-run

On Wed, Aug 1, 2012 at 1:33 PM, Keith Wiley <kw...@keithwiley.com> wrote:

> I know there is a lot of discussion about JVM reuse in Hadoop, but that
> usually refers to mappers running on the cluste nodesr.  I have a much
> different question.  I am running a Java program which at one point execs
> hadoop and that call sometimes fails in the fashion shown below.  Thus,
> this issue occurs entirely within the client machine (of course, I am
> currently running in pseudo-distributed mode which convolutes that point
> somewhat).  In other words, I successfully ran a Java program, but it
> failed to subsequently run *another* Java program (hadoop).  My
> interpretation of the hadoop startup scripts (the hadoop command itself for
> example) is that they run a second JVM in my scenario, and that they fail
> to allocate enough memory.
>
> Is there any way to run hadoop from within a JVM such that it reuses the
> local JVM?
>
> EXCEPTION: java.io.IOException: Cannot run program "hadoop":
> java.io.IOException: error=12, Cannot allocate memory
>     java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>     java.lang.Runtime.exec(Runtime.java:593)
>     java.lang.Runtime.exec(Runtime.java:466)
>     com.util.Shell.run(Shell.java:44)
>     com.exe.Foo.bar(Foo.java:107)
>     com.exe.Foo.run(Foo.java:205)
>     com.exe.Foo.main(Foo.java:227)
> Exception in thread "main" java.io.IOException: Cannot run program
> "hadoop": java.io.IOException: error=12, Cannot allocate memory
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>         at java.lang.Runtime.exec(Runtime.java:593)
>         at java.lang.Runtime.exec(Runtime.java:466)
>         at com.util.Shell.run(Shell.java:44)
>         at com.exe.Foo.bar(Foo.java:107)
>         at com.exe.Foo.run(Foo.java:205)
>         at com.exe.Foo.main(Foo.java:227)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>         ... 6 more
>
>
> ________________________________________________________________________________
> Keith Wiley     kwiley@keithwiley.com     keithwiley.com
> music.keithwiley.com
>
> "You can scratch an itch, but you can't itch a scratch. Furthermore, an
> itch can
> itch but a scratch can't scratch. Finally, a scratch can itch, but an itch
> can't
> scratch. All together this implies: He scratched the itch from the scratch
> that
> itched but would never itch the scratch from the itch that scratched."
>                                            --  Keith Wiley
>
> ________________________________________________________________________________
>
>