You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/03/26 23:37:46 UTC
Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
No, the “nice” is internal to Hadoop, apparently to make sure that the Hadoop infrastructure processes can’t be starved by the applications. Actually we figured out what was going on. There was a very large environment variable being set for the AM, and it exceeded the 132KB or so allowed for the total command-line-argument + environment space.
Cheers,
John
From: Azuryy [mailto:azuryyyu@gmail.com]
Sent: Wednesday, March 26, 2014 5:13 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
Glad to hear that :)
--
Wangda Tan
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 10:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Wangda Tan,
>
>
>
> Thanks for your reply! We did actually figure out where the problem was
> coming from, but this is a very helpful technique to know.
>
>
>
> John
>
>
>
>
>
> *From:* Wangda Tan [mailto:wheeleast@gmail.com]
> *Sent:* Wednesday, March 26, 2014 6:35 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Getting error message from AM container launch
>
>
>
> HI John,
>
> Typically, this is caused by somewhere in your program set "nice" as AM
> launching command. You can check the "real" script which YARN used to
> launch AM.
>
> You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
> all NMs to a larger value (like 600, 10 min), to make NMs don't remove
> temporary directory of a container as soon as the container get finished.
> You need restart NMs after you set.
>
> After that, you can re-run your program again, the script you can find
> should be
> <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
>
> You can verify the launch command if correct in the script.
>
> --
>
> Regards,
>
> Wangda Tan
>
>
>
> On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
>
> You used 'nice' in your app?
>
>
>
> Sent from my iPhone5s
>
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
>
>
>
> --
>
> Regards,
>
> Wangda
>
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
Glad to hear that :)
--
Wangda Tan
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 10:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Wangda Tan,
>
>
>
> Thanks for your reply! We did actually figure out where the problem was
> coming from, but this is a very helpful technique to know.
>
>
>
> John
>
>
>
>
>
> *From:* Wangda Tan [mailto:wheeleast@gmail.com]
> *Sent:* Wednesday, March 26, 2014 6:35 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Getting error message from AM container launch
>
>
>
> HI John,
>
> Typically, this is caused by somewhere in your program set "nice" as AM
> launching command. You can check the "real" script which YARN used to
> launch AM.
>
> You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
> all NMs to a larger value (like 600, 10 min), to make NMs don't remove
> temporary directory of a container as soon as the container get finished.
> You need restart NMs after you set.
>
> After that, you can re-run your program again, the script you can find
> should be
> <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
>
> You can verify the launch command if correct in the script.
>
> --
>
> Regards,
>
> Wangda Tan
>
>
>
> On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
>
> You used 'nice' in your app?
>
>
>
> Sent from my iPhone5s
>
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
>
>
>
> --
>
> Regards,
>
> Wangda
>
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
Glad to hear that :)
--
Wangda Tan
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 10:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Wangda Tan,
>
>
>
> Thanks for your reply! We did actually figure out where the problem was
> coming from, but this is a very helpful technique to know.
>
>
>
> John
>
>
>
>
>
> *From:* Wangda Tan [mailto:wheeleast@gmail.com]
> *Sent:* Wednesday, March 26, 2014 6:35 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Getting error message from AM container launch
>
>
>
> HI John,
>
> Typically, this is caused by somewhere in your program set "nice" as AM
> launching command. You can check the "real" script which YARN used to
> launch AM.
>
> You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
> all NMs to a larger value (like 600, 10 min), to make NMs don't remove
> temporary directory of a container as soon as the container get finished.
> You need restart NMs after you set.
>
> After that, you can re-run your program again, the script you can find
> should be
> <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
>
> You can verify the launch command if correct in the script.
>
> --
>
> Regards,
>
> Wangda Tan
>
>
>
> On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
>
> You used 'nice' in your app?
>
>
>
> Sent from my iPhone5s
>
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
>
>
>
> --
>
> Regards,
>
> Wangda
>
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
Glad to hear that :)
--
Wangda Tan
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 10:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Wangda Tan,
>
>
>
> Thanks for your reply! We did actually figure out where the problem was
> coming from, but this is a very helpful technique to know.
>
>
>
> John
>
>
>
>
>
> *From:* Wangda Tan [mailto:wheeleast@gmail.com]
> *Sent:* Wednesday, March 26, 2014 6:35 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Getting error message from AM container launch
>
>
>
> HI John,
>
> Typically, this is caused by somewhere in your program set "nice" as AM
> launching command. You can check the "real" script which YARN used to
> launch AM.
>
> You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
> all NMs to a larger value (like 600, 10 min), to make NMs don't remove
> temporary directory of a container as soon as the container get finished.
> You need restart NMs after you set.
>
> After that, you can re-run your program again, the script you can find
> should be
> <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
>
> You can verify the launch command if correct in the script.
>
> --
>
> Regards,
>
> Wangda Tan
>
>
>
> On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
>
> You used 'nice' in your app?
>
>
>
> Sent from my iPhone5s
>
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
>
>
>
> --
>
> Regards,
>
> Wangda
>
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
Wangda Tan,
Thanks for your reply! We did actually figure out where the problem was coming from, but this is a very helpful technique to know.
John
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Wednesday, March 26, 2014 6:35 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM launching command. You can check the "real" script which YARN used to launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on all NMs to a larger value (like 600, 10 min), to make NMs don't remove temporary directory of a container as soon as the container get finished. You need restart NMs after you set.
After that, you can re-run your program again, the script you can find should be <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com>> wrote:
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
--
Regards,
Wangda
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
Wangda Tan,
Thanks for your reply! We did actually figure out where the problem was coming from, but this is a very helpful technique to know.
John
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Wednesday, March 26, 2014 6:35 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM launching command. You can check the "real" script which YARN used to launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on all NMs to a larger value (like 600, 10 min), to make NMs don't remove temporary directory of a container as soon as the container get finished. You need restart NMs after you set.
After that, you can re-run your program again, the script you can find should be <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com>> wrote:
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
--
Regards,
Wangda
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
Wangda Tan,
Thanks for your reply! We did actually figure out where the problem was coming from, but this is a very helpful technique to know.
John
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Wednesday, March 26, 2014 6:35 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM launching command. You can check the "real" script which YARN used to launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on all NMs to a larger value (like 600, 10 min), to make NMs don't remove temporary directory of a container as soon as the container get finished. You need restart NMs after you set.
After that, you can re-run your program again, the script you can find should be <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com>> wrote:
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
--
Regards,
Wangda
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
Wangda Tan,
Thanks for your reply! We did actually figure out where the problem was coming from, but this is a very helpful technique to know.
John
From: Wangda Tan [mailto:wheeleast@gmail.com]
Sent: Wednesday, March 26, 2014 6:35 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM launching command. You can check the "real" script which YARN used to launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on all NMs to a larger value (like 600, 10 min), to make NMs don't remove temporary directory of a container as soon as the container get finished. You need restart NMs after you set.
After that, you can re-run your program again, the script you can find should be <host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com>> wrote:
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
--
Regards,
Wangda
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM
launching command. You can check the "real" script which YARN used to
launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
all NMs to a larger value (like 600, 10 min), to make NMs don't remove
temporary directory of a container as soon as the container get finished.
You need restart NMs after you set.
After that, you can re-run your program again, the script you can find
should be
<host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
> You used 'nice' in your app?
>
>
> Sent from my iPhone5s
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
--
Regards,
Wangda
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM
launching command. You can check the "real" script which YARN used to
launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
all NMs to a larger value (like 600, 10 min), to make NMs don't remove
temporary directory of a container as soon as the container get finished.
You need restart NMs after you set.
After that, you can re-run your program again, the script you can find
should be
<host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
> You used 'nice' in your app?
>
>
> Sent from my iPhone5s
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
--
Regards,
Wangda
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
No, the “nice” is internal to Hadoop, apparently to make sure that the Hadoop infrastructure processes can’t be starved by the applications. Actually we figured out what was going on. There was a very large environment variable being set for the AM, and it exceeded the 132KB or so allowed for the total command-line-argument + environment space.
Cheers,
John
From: Azuryy [mailto:azuryyyu@gmail.com]
Sent: Wednesday, March 26, 2014 5:13 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
No, the “nice” is internal to Hadoop, apparently to make sure that the Hadoop infrastructure processes can’t be starved by the applications. Actually we figured out what was going on. There was a very large environment variable being set for the AM, and it exceeded the 132KB or so allowed for the total command-line-argument + environment space.
Cheers,
John
From: Azuryy [mailto:azuryyyu@gmail.com]
Sent: Wednesday, March 26, 2014 5:13 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM
launching command. You can check the "real" script which YARN used to
launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
all NMs to a larger value (like 600, 10 min), to make NMs don't remove
temporary directory of a container as soon as the container get finished.
You need restart NMs after you set.
After that, you can re-run your program again, the script you can find
should be
<host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
> You used 'nice' in your app?
>
>
> Sent from my iPhone5s
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
--
Regards,
Wangda
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
No, the “nice” is internal to Hadoop, apparently to make sure that the Hadoop infrastructure processes can’t be starved by the applications. Actually we figured out what was going on. There was a very large environment variable being set for the AM, and it exceeded the 132KB or so allowed for the total command-line-argument + environment space.
Cheers,
John
From: Azuryy [mailto:azuryyyu@gmail.com]
Sent: Wednesday, March 26, 2014 5:13 PM
To: user@hadoop.apache.org
Subject: Re: Getting error message from AM container launch
You used 'nice' in your app?
Sent from my iPhone5s
On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net>> wrote:
On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
Re: Getting error message from AM container launch
Posted by Wangda Tan <wh...@gmail.com>.
HI John,
Typically, this is caused by somewhere in your program set "nice" as AM
launching command. You can check the "real" script which YARN used to
launch AM.
You need set "yarn.nodemanager.delete.debug-delay-sec" in yarn-site.xml on
all NMs to a larger value (like 600, 10 min), to make NMs don't remove
temporary directory of a container as soon as the container get finished.
You need restart NMs after you set.
After that, you can re-run your program again, the script you can find
should be
<host-of-AM>:/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/<app-id>/<container-id>/launch_container.sh.
You can verify the launch command if correct in the script.
--
Regards,
Wangda Tan
On Thu, Mar 27, 2014 at 7:12 AM, Azuryy <az...@gmail.com> wrote:
> You used 'nice' in your app?
>
>
> Sent from my iPhone5s
>
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read
> about similar issues showing when the environment exceeds 132KB, but we
> aren’t putting anything significant in the environment.
>
> John
>
>
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:41 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: Getting error message from AM container launch
>
>
>
> We do have a fairly long container command-line. Not huge, around 200
> characters.
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net<jo...@redpoint.net>]
>
> *Sent:* Wednesday, March 26, 2014 4:38 PM
> *To:* user@hadoop.apache.org
> *Subject:* Getting error message from AM container launch
>
>
>
> Running a non-MapReduce YARN application, one of the containers launched
> by the AM is failing with an error message I’ve never seen. Any ideas?
> I’m not sure who exactly is running “nice” or why its argument list would
> be too long.
>
> Thanks
>
> john
>
>
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode:
> 0 due to: Exception from container-launch:
>
> java.io.IOException: Cannot run program ""nice"" (in directory
> ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""):
> java.io.IOException: error=7, Argument list too long
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument
> list too long
>
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>
> ... 11 more
>
>
>
>
--
Regards,
Wangda
Re: Getting error message from AM container launch
Posted by Azuryy <az...@gmail.com>.
You used 'nice' in your app?
Sent from my iPhone5s
> On 2014��3��27��, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I��ve read about similar issues showing when the environment exceeds 132KB, but we aren��t putting anything significant in the environment.
> John
>
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:41 PM
> To: user@hadoop.apache.org
> Subject: RE: Getting error message from AM container launch
>
> We do have a fairly long container command-line. Not huge, around 200 characters.
> John
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:38 PM
> To: user@hadoop.apache.org
> Subject: Getting error message from AM container launch
>
> Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I��ve never seen. Any ideas? I��m not sure who exactly is running ��nice�� or why its argument list would be too long.
> Thanks
> john
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
> java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 11 more
>
Re: Getting error message from AM container launch
Posted by Azuryy <az...@gmail.com>.
You used 'nice' in your app?
Sent from my iPhone5s
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
> John
>
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:41 PM
> To: user@hadoop.apache.org
> Subject: RE: Getting error message from AM container launch
>
> We do have a fairly long container command-line. Not huge, around 200 characters.
> John
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:38 PM
> To: user@hadoop.apache.org
> Subject: Getting error message from AM container launch
>
> Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
> Thanks
> john
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
> java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 11 more
>
Re: Getting error message from AM container launch
Posted by Azuryy <az...@gmail.com>.
You used 'nice' in your app?
Sent from my iPhone5s
> On 2014年3月27日, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I’ve read about similar issues showing when the environment exceeds 132KB, but we aren’t putting anything significant in the environment.
> John
>
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:41 PM
> To: user@hadoop.apache.org
> Subject: RE: Getting error message from AM container launch
>
> We do have a fairly long container command-line. Not huge, around 200 characters.
> John
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:38 PM
> To: user@hadoop.apache.org
> Subject: Getting error message from AM container launch
>
> Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I’ve never seen. Any ideas? I’m not sure who exactly is running “nice” or why its argument list would be too long.
> Thanks
> john
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
> java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 11 more
>
Re: Getting error message from AM container launch
Posted by Azuryy <az...@gmail.com>.
You used 'nice' in your app?
Sent from my iPhone5s
> On 2014��3��27��, at 6:55, John Lilley <jo...@redpoint.net> wrote:
>
> On further examination they appear to be 369 characters long. I��ve read about similar issues showing when the environment exceeds 132KB, but we aren��t putting anything significant in the environment.
> John
>
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:41 PM
> To: user@hadoop.apache.org
> Subject: RE: Getting error message from AM container launch
>
> We do have a fairly long container command-line. Not huge, around 200 characters.
> John
>
> From: John Lilley [mailto:john.lilley@redpoint.net]
> Sent: Wednesday, March 26, 2014 4:38 PM
> To: user@hadoop.apache.org
> Subject: Getting error message from AM container launch
>
> Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I��ve never seen. Any ideas? I��m not sure who exactly is running ��nice�� or why its argument list would be too long.
> Thanks
> john
>
> Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
> java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 11 more
>
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
On further examination they appear to be 369 characters long. I've read about similar issues showing when the environment exceeds 132KB, but we aren't putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
On further examination they appear to be 369 characters long. I've read about similar issues showing when the environment exceeds 132KB, but we aren't putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
On further examination they appear to be 369 characters long. I've read about similar issues showing when the environment exceeds 132KB, but we aren't putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
On further examination they appear to be 369 characters long. I've read about similar issues showing when the environment exceeds 132KB, but we aren't putting anything significant in the environment.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:41 PM
To: user@hadoop.apache.org
Subject: RE: Getting error message from AM container launch
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
RE: Getting error message from AM container launch
Posted by John Lilley <jo...@redpoint.net>.
We do have a fairly long container command-line. Not huge, around 200 characters.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Wednesday, March 26, 2014 4:38 PM
To: user@hadoop.apache.org
Subject: Getting error message from AM container launch
Running a non-MapReduce YARN application, one of the containers launched by the AM is failing with an error message I've never seen. Any ideas? I'm not sure who exactly is running "nice" or why its argument list would be too long.
Thanks
john
Container for appattempt_1395755163053_0030_000001 exited with exitCode: 0 due to: Exception from container-launch:
java.io.IOException: Cannot run program ""nice"" (in directory ""/ephemeral02/hadoop/yarn/local/usercache/SYSTEM/appcache/application_1395755163053_0030/container_1395755163053_0030_01_000001""): java.io.IOException: error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:407)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more