You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Curtis Ullerich <cu...@gmail.com> on 2013/06/18 04:49:09 UTC
Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in standalone mode
by launching the pi estimator example with this command:
hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
"-Dyarn.app.mapreduce.am.command-opts=-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10
and then attaching a debugger to port 8000 using Eclipse. This doesn't work
with the DistributedShell example, presumably because it's not configurable
with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in
yarn-default.xml, I don't see an equivalent parameter. For learning
purposes, how can I debug the DistributedShell example (and other AMs)?
Thanks!
Curtis
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
For others looking for a solution, here's one way to do it.
You can add the arg directly to the code in hadoop-yarn-applications-
distributedshell/src/main/java/org/apache/hadoop/yarn/
applications/distributedshell/Client.java
Rather than referencing line numbers (as there are frequent commits to
this), find the variable 'vargs' inside Client.run(). The first item added
to that list is the qualified path to /bin/java.
vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
Add this line below that:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1337");
(This will be dependent on how you're building/running)
>From the distributedshell directory, run mvn clean package -DskipTests; mvn
install -DskipTests. You may need to copy the jar from target into
hadoop-dist/... so it's properly available on Hadoop's classpath.
Run the jar like normal
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
In Eclipse, set up a new debug configuration for a remote Java application.
Select hadoop-yarn-applications-distributedshell as the project. Use
localhost as the Host and 1337 as the port., with connection type of
Standard (Socket Attach). Hit 'Debug' and Eclipse should connect to the
running Application Master. Be sure to set a breakpoint inside main (or
wherever) in the distributedshell project, which must be present in your
Eclipse workspace. Now you can step through the AM.
Let me know of any confusion or questions.
Best,
Curtis
On Tue, Jun 18, 2013 at 3:10 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> I'm under the impression that this will launch hadoop (so, the
> ResourceManager?) with those options, and not the ApplicationMaster. Is
> that incorrect?
>
>
> On Tue, Jun 18, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> If distributed shell is running as an unmanaged AM then you should set
>> the debug flags for the 'hadoop jar' invocation, doing an 'export
>> HADOOP_OPTS=....' with the debug flags would do.
>>
>> Thx
>>
>>
>> On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <curtullerich@gmail.com
>> > wrote:
>>
>>> Update: It looks like I could add the flag at line 515 of
>>> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>>> (package org.apache.hadoop.yarn.applications.distributedshell).
>>>
>>> I tried this:
>>> vargs.add("-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>>>
>>> I recompiled the jar and launched it with
>>> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
>>> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
>>> whoami
>>>
>>> I added a logging statement to make sure the change took effect. It
>>> prints, but it doesn't seem to halt like the MR example (because of the
>>> suspend=y option) even after launching the debugger. Why might this be?
>>>
>>> Thanks!
>>> Curtis
>>>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <curtullerich@gmail.com
>>> > wrote:
>>>
>>>> Hi Devaraj,
>>>>
>>>> That's what I'm not sure how to do. I want to debug by connecting
>>>> remotely, and I'm not sure how to configure that in the code. I haven't
>>>> found anything telling in the docs/source. Can you point me in the right
>>>> direction?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>>
>>>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com>wrote:
>>>>
>>>>> Hi Curtis, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>>>
>>>>> ** **
>>>>>
>>>>> If you want to dump out debug information then you can make use of the
>>>>> debug option of DistributedShell application. If you want to debug by
>>>>> connecting remotely, you need to update the DS application code accordingly.
>>>>> ****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks****
>>>>>
>>>>> Devaraj K****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>>>> *Sent:* 18 June 2013 08:19
>>>>> *To:* user@hadoop.apache.org
>>>>> *Subject:* Debugging YARN AM****
>>>>>
>>>>> ** **
>>>>>
>>>>> Hi all, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>>>> mode by launching the pi estimator example with this command:****
>>>>>
>>>>> ** **
>>>>>
>>>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10**
>>>>> **
>>>>>
>>>>> ** **
>>>>>
>>>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>>>> work with the DistributedShell example, presumably because it's not
>>>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>>>> learning purposes, how can I debug the DistributedShell example (and other
>>>>> AMs)?****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks!****
>>>>>
>>>>> ** **
>>>>>
>>>>> Curtis****
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
For others looking for a solution, here's one way to do it.
You can add the arg directly to the code in hadoop-yarn-applications-
distributedshell/src/main/java/org/apache/hadoop/yarn/
applications/distributedshell/Client.java
Rather than referencing line numbers (as there are frequent commits to
this), find the variable 'vargs' inside Client.run(). The first item added
to that list is the qualified path to /bin/java.
vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
Add this line below that:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1337");
(This will be dependent on how you're building/running)
>From the distributedshell directory, run mvn clean package -DskipTests; mvn
install -DskipTests. You may need to copy the jar from target into
hadoop-dist/... so it's properly available on Hadoop's classpath.
Run the jar like normal
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
In Eclipse, set up a new debug configuration for a remote Java application.
Select hadoop-yarn-applications-distributedshell as the project. Use
localhost as the Host and 1337 as the port., with connection type of
Standard (Socket Attach). Hit 'Debug' and Eclipse should connect to the
running Application Master. Be sure to set a breakpoint inside main (or
wherever) in the distributedshell project, which must be present in your
Eclipse workspace. Now you can step through the AM.
Let me know of any confusion or questions.
Best,
Curtis
On Tue, Jun 18, 2013 at 3:10 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> I'm under the impression that this will launch hadoop (so, the
> ResourceManager?) with those options, and not the ApplicationMaster. Is
> that incorrect?
>
>
> On Tue, Jun 18, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> If distributed shell is running as an unmanaged AM then you should set
>> the debug flags for the 'hadoop jar' invocation, doing an 'export
>> HADOOP_OPTS=....' with the debug flags would do.
>>
>> Thx
>>
>>
>> On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <curtullerich@gmail.com
>> > wrote:
>>
>>> Update: It looks like I could add the flag at line 515 of
>>> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>>> (package org.apache.hadoop.yarn.applications.distributedshell).
>>>
>>> I tried this:
>>> vargs.add("-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>>>
>>> I recompiled the jar and launched it with
>>> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
>>> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
>>> whoami
>>>
>>> I added a logging statement to make sure the change took effect. It
>>> prints, but it doesn't seem to halt like the MR example (because of the
>>> suspend=y option) even after launching the debugger. Why might this be?
>>>
>>> Thanks!
>>> Curtis
>>>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <curtullerich@gmail.com
>>> > wrote:
>>>
>>>> Hi Devaraj,
>>>>
>>>> That's what I'm not sure how to do. I want to debug by connecting
>>>> remotely, and I'm not sure how to configure that in the code. I haven't
>>>> found anything telling in the docs/source. Can you point me in the right
>>>> direction?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>>
>>>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com>wrote:
>>>>
>>>>> Hi Curtis, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>>>
>>>>> ** **
>>>>>
>>>>> If you want to dump out debug information then you can make use of the
>>>>> debug option of DistributedShell application. If you want to debug by
>>>>> connecting remotely, you need to update the DS application code accordingly.
>>>>> ****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks****
>>>>>
>>>>> Devaraj K****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>>>> *Sent:* 18 June 2013 08:19
>>>>> *To:* user@hadoop.apache.org
>>>>> *Subject:* Debugging YARN AM****
>>>>>
>>>>> ** **
>>>>>
>>>>> Hi all, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>>>> mode by launching the pi estimator example with this command:****
>>>>>
>>>>> ** **
>>>>>
>>>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10**
>>>>> **
>>>>>
>>>>> ** **
>>>>>
>>>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>>>> work with the DistributedShell example, presumably because it's not
>>>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>>>> learning purposes, how can I debug the DistributedShell example (and other
>>>>> AMs)?****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks!****
>>>>>
>>>>> ** **
>>>>>
>>>>> Curtis****
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
For others looking for a solution, here's one way to do it.
You can add the arg directly to the code in hadoop-yarn-applications-
distributedshell/src/main/java/org/apache/hadoop/yarn/
applications/distributedshell/Client.java
Rather than referencing line numbers (as there are frequent commits to
this), find the variable 'vargs' inside Client.run(). The first item added
to that list is the qualified path to /bin/java.
vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
Add this line below that:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1337");
(This will be dependent on how you're building/running)
>From the distributedshell directory, run mvn clean package -DskipTests; mvn
install -DskipTests. You may need to copy the jar from target into
hadoop-dist/... so it's properly available on Hadoop's classpath.
Run the jar like normal
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
In Eclipse, set up a new debug configuration for a remote Java application.
Select hadoop-yarn-applications-distributedshell as the project. Use
localhost as the Host and 1337 as the port., with connection type of
Standard (Socket Attach). Hit 'Debug' and Eclipse should connect to the
running Application Master. Be sure to set a breakpoint inside main (or
wherever) in the distributedshell project, which must be present in your
Eclipse workspace. Now you can step through the AM.
Let me know of any confusion or questions.
Best,
Curtis
On Tue, Jun 18, 2013 at 3:10 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> I'm under the impression that this will launch hadoop (so, the
> ResourceManager?) with those options, and not the ApplicationMaster. Is
> that incorrect?
>
>
> On Tue, Jun 18, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> If distributed shell is running as an unmanaged AM then you should set
>> the debug flags for the 'hadoop jar' invocation, doing an 'export
>> HADOOP_OPTS=....' with the debug flags would do.
>>
>> Thx
>>
>>
>> On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <curtullerich@gmail.com
>> > wrote:
>>
>>> Update: It looks like I could add the flag at line 515 of
>>> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>>> (package org.apache.hadoop.yarn.applications.distributedshell).
>>>
>>> I tried this:
>>> vargs.add("-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>>>
>>> I recompiled the jar and launched it with
>>> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
>>> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
>>> whoami
>>>
>>> I added a logging statement to make sure the change took effect. It
>>> prints, but it doesn't seem to halt like the MR example (because of the
>>> suspend=y option) even after launching the debugger. Why might this be?
>>>
>>> Thanks!
>>> Curtis
>>>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <curtullerich@gmail.com
>>> > wrote:
>>>
>>>> Hi Devaraj,
>>>>
>>>> That's what I'm not sure how to do. I want to debug by connecting
>>>> remotely, and I'm not sure how to configure that in the code. I haven't
>>>> found anything telling in the docs/source. Can you point me in the right
>>>> direction?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>>
>>>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com>wrote:
>>>>
>>>>> Hi Curtis, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>>>
>>>>> ** **
>>>>>
>>>>> If you want to dump out debug information then you can make use of the
>>>>> debug option of DistributedShell application. If you want to debug by
>>>>> connecting remotely, you need to update the DS application code accordingly.
>>>>> ****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks****
>>>>>
>>>>> Devaraj K****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>>>> *Sent:* 18 June 2013 08:19
>>>>> *To:* user@hadoop.apache.org
>>>>> *Subject:* Debugging YARN AM****
>>>>>
>>>>> ** **
>>>>>
>>>>> Hi all, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>>>> mode by launching the pi estimator example with this command:****
>>>>>
>>>>> ** **
>>>>>
>>>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10**
>>>>> **
>>>>>
>>>>> ** **
>>>>>
>>>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>>>> work with the DistributedShell example, presumably because it's not
>>>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>>>> learning purposes, how can I debug the DistributedShell example (and other
>>>>> AMs)?****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks!****
>>>>>
>>>>> ** **
>>>>>
>>>>> Curtis****
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
For others looking for a solution, here's one way to do it.
You can add the arg directly to the code in hadoop-yarn-applications-
distributedshell/src/main/java/org/apache/hadoop/yarn/
applications/distributedshell/Client.java
Rather than referencing line numbers (as there are frequent commits to
this), find the variable 'vargs' inside Client.run(). The first item added
to that list is the qualified path to /bin/java.
vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
Add this line below that:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1337");
(This will be dependent on how you're building/running)
>From the distributedshell directory, run mvn clean package -DskipTests; mvn
install -DskipTests. You may need to copy the jar from target into
hadoop-dist/... so it's properly available on Hadoop's classpath.
Run the jar like normal
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
In Eclipse, set up a new debug configuration for a remote Java application.
Select hadoop-yarn-applications-distributedshell as the project. Use
localhost as the Host and 1337 as the port., with connection type of
Standard (Socket Attach). Hit 'Debug' and Eclipse should connect to the
running Application Master. Be sure to set a breakpoint inside main (or
wherever) in the distributedshell project, which must be present in your
Eclipse workspace. Now you can step through the AM.
Let me know of any confusion or questions.
Best,
Curtis
On Tue, Jun 18, 2013 at 3:10 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> I'm under the impression that this will launch hadoop (so, the
> ResourceManager?) with those options, and not the ApplicationMaster. Is
> that incorrect?
>
>
> On Tue, Jun 18, 2013 at 2:29 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> If distributed shell is running as an unmanaged AM then you should set
>> the debug flags for the 'hadoop jar' invocation, doing an 'export
>> HADOOP_OPTS=....' with the debug flags would do.
>>
>> Thx
>>
>>
>> On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <curtullerich@gmail.com
>> > wrote:
>>
>>> Update: It looks like I could add the flag at line 515 of
>>> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>>> (package org.apache.hadoop.yarn.applications.distributedshell).
>>>
>>> I tried this:
>>> vargs.add("-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>>>
>>> I recompiled the jar and launched it with
>>> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
>>> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
>>> whoami
>>>
>>> I added a logging statement to make sure the change took effect. It
>>> prints, but it doesn't seem to halt like the MR example (because of the
>>> suspend=y option) even after launching the debugger. Why might this be?
>>>
>>> Thanks!
>>> Curtis
>>>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <curtullerich@gmail.com
>>> > wrote:
>>>
>>>> Hi Devaraj,
>>>>
>>>> That's what I'm not sure how to do. I want to debug by connecting
>>>> remotely, and I'm not sure how to configure that in the code. I haven't
>>>> found anything telling in the docs/source. Can you point me in the right
>>>> direction?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>>
>>>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com>wrote:
>>>>
>>>>> Hi Curtis, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>>>
>>>>> ** **
>>>>>
>>>>> If you want to dump out debug information then you can make use of the
>>>>> debug option of DistributedShell application. If you want to debug by
>>>>> connecting remotely, you need to update the DS application code accordingly.
>>>>> ****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks****
>>>>>
>>>>> Devaraj K****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>>>> *Sent:* 18 June 2013 08:19
>>>>> *To:* user@hadoop.apache.org
>>>>> *Subject:* Debugging YARN AM****
>>>>>
>>>>> ** **
>>>>>
>>>>> Hi all, ****
>>>>>
>>>>> ** **
>>>>>
>>>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>>>> mode by launching the pi estimator example with this command:****
>>>>>
>>>>> ** **
>>>>>
>>>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10**
>>>>> **
>>>>>
>>>>> ** **
>>>>>
>>>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>>>> work with the DistributedShell example, presumably because it's not
>>>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>>>> learning purposes, how can I debug the DistributedShell example (and other
>>>>> AMs)?****
>>>>>
>>>>> ** **
>>>>>
>>>>> Thanks!****
>>>>>
>>>>> ** **
>>>>>
>>>>> Curtis****
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Debugging YARN AM
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
If distributed shell is running as an unmanaged AM then you should set the
debug flags for the 'hadoop jar' invocation, doing an 'export
HADOOP_OPTS=....' with the debug flags would do.
Thx
On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> Update: It looks like I could add the flag at line 515 of
> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> (package org.apache.hadoop.yarn.applications.distributedshell).
>
> I tried this:
> vargs.add("-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>
> I recompiled the jar and launched it with
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
> whoami
>
> I added a logging statement to make sure the change took effect. It
> prints, but it doesn't seem to halt like the MR example (because of the
> suspend=y option) even after launching the debugger. Why might this be?
>
> Thanks!
> Curtis
>
>
>
> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
>
>> Hi Devaraj,
>>
>> That's what I'm not sure how to do. I want to debug by connecting
>> remotely, and I'm not sure how to configure that in the code. I haven't
>> found anything telling in the docs/source. Can you point me in the right
>> direction?
>>
>> Thanks,
>> Curtis
>>
>>
>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>>
>>> Hi Curtis, ****
>>>
>>> ** **
>>>
>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>
>>> ** **
>>>
>>> If you want to dump out debug information then you can make use of the
>>> debug option of DistributedShell application. If you want to debug by
>>> connecting remotely, you need to update the DS application code accordingly.
>>> ****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> Devaraj K****
>>>
>>> ** **
>>>
>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>> *Sent:* 18 June 2013 08:19
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Debugging YARN AM****
>>>
>>> ** **
>>>
>>> Hi all, ****
>>>
>>> ** **
>>>
>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>> mode by launching the pi estimator example with this command:****
>>>
>>> ** **
>>>
>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>>
>>> ** **
>>>
>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>> work with the DistributedShell example, presumably because it's not
>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>> learning purposes, how can I debug the DistributedShell example (and other
>>> AMs)?****
>>>
>>> ** **
>>>
>>> Thanks!****
>>>
>>> ** **
>>>
>>> Curtis****
>>>
>>
>>
>
--
Alejandro
Re: Debugging YARN AM
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
If distributed shell is running as an unmanaged AM then you should set the
debug flags for the 'hadoop jar' invocation, doing an 'export
HADOOP_OPTS=....' with the debug flags would do.
Thx
On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> Update: It looks like I could add the flag at line 515 of
> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> (package org.apache.hadoop.yarn.applications.distributedshell).
>
> I tried this:
> vargs.add("-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>
> I recompiled the jar and launched it with
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
> whoami
>
> I added a logging statement to make sure the change took effect. It
> prints, but it doesn't seem to halt like the MR example (because of the
> suspend=y option) even after launching the debugger. Why might this be?
>
> Thanks!
> Curtis
>
>
>
> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
>
>> Hi Devaraj,
>>
>> That's what I'm not sure how to do. I want to debug by connecting
>> remotely, and I'm not sure how to configure that in the code. I haven't
>> found anything telling in the docs/source. Can you point me in the right
>> direction?
>>
>> Thanks,
>> Curtis
>>
>>
>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>>
>>> Hi Curtis, ****
>>>
>>> ** **
>>>
>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>
>>> ** **
>>>
>>> If you want to dump out debug information then you can make use of the
>>> debug option of DistributedShell application. If you want to debug by
>>> connecting remotely, you need to update the DS application code accordingly.
>>> ****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> Devaraj K****
>>>
>>> ** **
>>>
>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>> *Sent:* 18 June 2013 08:19
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Debugging YARN AM****
>>>
>>> ** **
>>>
>>> Hi all, ****
>>>
>>> ** **
>>>
>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>> mode by launching the pi estimator example with this command:****
>>>
>>> ** **
>>>
>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>>
>>> ** **
>>>
>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>> work with the DistributedShell example, presumably because it's not
>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>> learning purposes, how can I debug the DistributedShell example (and other
>>> AMs)?****
>>>
>>> ** **
>>>
>>> Thanks!****
>>>
>>> ** **
>>>
>>> Curtis****
>>>
>>
>>
>
--
Alejandro
Re: Debugging YARN AM
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
If distributed shell is running as an unmanaged AM then you should set the
debug flags for the 'hadoop jar' invocation, doing an 'export
HADOOP_OPTS=....' with the debug flags would do.
Thx
On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> Update: It looks like I could add the flag at line 515 of
> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> (package org.apache.hadoop.yarn.applications.distributedshell).
>
> I tried this:
> vargs.add("-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>
> I recompiled the jar and launched it with
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
> whoami
>
> I added a logging statement to make sure the change took effect. It
> prints, but it doesn't seem to halt like the MR example (because of the
> suspend=y option) even after launching the debugger. Why might this be?
>
> Thanks!
> Curtis
>
>
>
> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
>
>> Hi Devaraj,
>>
>> That's what I'm not sure how to do. I want to debug by connecting
>> remotely, and I'm not sure how to configure that in the code. I haven't
>> found anything telling in the docs/source. Can you point me in the right
>> direction?
>>
>> Thanks,
>> Curtis
>>
>>
>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>>
>>> Hi Curtis, ****
>>>
>>> ** **
>>>
>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>
>>> ** **
>>>
>>> If you want to dump out debug information then you can make use of the
>>> debug option of DistributedShell application. If you want to debug by
>>> connecting remotely, you need to update the DS application code accordingly.
>>> ****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> Devaraj K****
>>>
>>> ** **
>>>
>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>> *Sent:* 18 June 2013 08:19
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Debugging YARN AM****
>>>
>>> ** **
>>>
>>> Hi all, ****
>>>
>>> ** **
>>>
>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>> mode by launching the pi estimator example with this command:****
>>>
>>> ** **
>>>
>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>>
>>> ** **
>>>
>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>> work with the DistributedShell example, presumably because it's not
>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>> learning purposes, how can I debug the DistributedShell example (and other
>>> AMs)?****
>>>
>>> ** **
>>>
>>> Thanks!****
>>>
>>> ** **
>>>
>>> Curtis****
>>>
>>
>>
>
--
Alejandro
Re: Debugging YARN AM
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
If distributed shell is running as an unmanaged AM then you should set the
debug flags for the 'hadoop jar' invocation, doing an 'export
HADOOP_OPTS=....' with the debug flags would do.
Thx
On Tue, Jun 18, 2013 at 12:32 PM, Curtis Ullerich <cu...@gmail.com>wrote:
> Update: It looks like I could add the flag at line 515 of
> hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
> (package org.apache.hadoop.yarn.applications.distributedshell).
>
> I tried this:
> vargs.add("-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
>
> I recompiled the jar and launched it with
> hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
> whoami
>
> I added a logging statement to make sure the change took effect. It
> prints, but it doesn't seem to halt like the MR example (because of the
> suspend=y option) even after launching the debugger. Why might this be?
>
> Thanks!
> Curtis
>
>
>
> On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
>
>> Hi Devaraj,
>>
>> That's what I'm not sure how to do. I want to debug by connecting
>> remotely, and I'm not sure how to configure that in the code. I haven't
>> found anything telling in the docs/source. Can you point me in the right
>> direction?
>>
>> Thanks,
>> Curtis
>>
>>
>> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>>
>>> Hi Curtis, ****
>>>
>>> ** **
>>>
>>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>>
>>> ** **
>>>
>>> If you want to dump out debug information then you can make use of the
>>> debug option of DistributedShell application. If you want to debug by
>>> connecting remotely, you need to update the DS application code accordingly.
>>> ****
>>>
>>> ** **
>>>
>>> Thanks****
>>>
>>> Devaraj K****
>>>
>>> ** **
>>>
>>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>>> *Sent:* 18 June 2013 08:19
>>> *To:* user@hadoop.apache.org
>>> *Subject:* Debugging YARN AM****
>>>
>>> ** **
>>>
>>> Hi all, ****
>>>
>>> ** **
>>>
>>> I can successfully debug the MapReduce ApplicationMaster in standalone
>>> mode by launching the pi estimator example with this command:****
>>>
>>> ** **
>>>
>>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>>
>>> ** **
>>>
>>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>>> work with the DistributedShell example, presumably because it's not
>>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>>> learning purposes, how can I debug the DistributedShell example (and other
>>> AMs)?****
>>>
>>> ** **
>>>
>>> Thanks!****
>>>
>>> ** **
>>>
>>> Curtis****
>>>
>>
>>
>
--
Alejandro
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
Update: It looks like I could add the flag at line 515 of
hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
(package org.apache.hadoop.yarn.applications.distributedshell).
I tried this:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
I recompiled the jar and launched it with
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
I added a logging statement to make sure the change took effect. It prints,
but it doesn't seem to halt like the MR example (because of the suspend=y
option) even after launching the debugger. Why might this be?
Thanks!
Curtis
On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
> Hi Devaraj,
>
> That's what I'm not sure how to do. I want to debug by connecting
> remotely, and I'm not sure how to configure that in the code. I haven't
> found anything telling in the docs/source. Can you point me in the right
> direction?
>
> Thanks,
> Curtis
>
>
> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>
>> Hi Curtis, ****
>>
>> ** **
>>
>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>
>> ** **
>>
>> If you want to dump out debug information then you can make use of the
>> debug option of DistributedShell application. If you want to debug by
>> connecting remotely, you need to update the DS application code accordingly.
>> ****
>>
>> ** **
>>
>> Thanks****
>>
>> Devaraj K****
>>
>> ** **
>>
>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>> *Sent:* 18 June 2013 08:19
>> *To:* user@hadoop.apache.org
>> *Subject:* Debugging YARN AM****
>>
>> ** **
>>
>> Hi all, ****
>>
>> ** **
>>
>> I can successfully debug the MapReduce ApplicationMaster in standalone
>> mode by launching the pi estimator example with this command:****
>>
>> ** **
>>
>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>
>> ** **
>>
>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>> work with the DistributedShell example, presumably because it's not
>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>> learning purposes, how can I debug the DistributedShell example (and other
>> AMs)?****
>>
>> ** **
>>
>> Thanks!****
>>
>> ** **
>>
>> Curtis****
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
Update: It looks like I could add the flag at line 515 of
hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
(package org.apache.hadoop.yarn.applications.distributedshell).
I tried this:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
I recompiled the jar and launched it with
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
I added a logging statement to make sure the change took effect. It prints,
but it doesn't seem to halt like the MR example (because of the suspend=y
option) even after launching the debugger. Why might this be?
Thanks!
Curtis
On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
> Hi Devaraj,
>
> That's what I'm not sure how to do. I want to debug by connecting
> remotely, and I'm not sure how to configure that in the code. I haven't
> found anything telling in the docs/source. Can you point me in the right
> direction?
>
> Thanks,
> Curtis
>
>
> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>
>> Hi Curtis, ****
>>
>> ** **
>>
>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>
>> ** **
>>
>> If you want to dump out debug information then you can make use of the
>> debug option of DistributedShell application. If you want to debug by
>> connecting remotely, you need to update the DS application code accordingly.
>> ****
>>
>> ** **
>>
>> Thanks****
>>
>> Devaraj K****
>>
>> ** **
>>
>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>> *Sent:* 18 June 2013 08:19
>> *To:* user@hadoop.apache.org
>> *Subject:* Debugging YARN AM****
>>
>> ** **
>>
>> Hi all, ****
>>
>> ** **
>>
>> I can successfully debug the MapReduce ApplicationMaster in standalone
>> mode by launching the pi estimator example with this command:****
>>
>> ** **
>>
>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>
>> ** **
>>
>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>> work with the DistributedShell example, presumably because it's not
>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>> learning purposes, how can I debug the DistributedShell example (and other
>> AMs)?****
>>
>> ** **
>>
>> Thanks!****
>>
>> ** **
>>
>> Curtis****
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
Update: It looks like I could add the flag at line 515 of
hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
(package org.apache.hadoop.yarn.applications.distributedshell).
I tried this:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
I recompiled the jar and launched it with
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
I added a logging statement to make sure the change took effect. It prints,
but it doesn't seem to halt like the MR example (because of the suspend=y
option) even after launching the debugger. Why might this be?
Thanks!
Curtis
On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
> Hi Devaraj,
>
> That's what I'm not sure how to do. I want to debug by connecting
> remotely, and I'm not sure how to configure that in the code. I haven't
> found anything telling in the docs/source. Can you point me in the right
> direction?
>
> Thanks,
> Curtis
>
>
> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>
>> Hi Curtis, ****
>>
>> ** **
>>
>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>
>> ** **
>>
>> If you want to dump out debug information then you can make use of the
>> debug option of DistributedShell application. If you want to debug by
>> connecting remotely, you need to update the DS application code accordingly.
>> ****
>>
>> ** **
>>
>> Thanks****
>>
>> Devaraj K****
>>
>> ** **
>>
>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>> *Sent:* 18 June 2013 08:19
>> *To:* user@hadoop.apache.org
>> *Subject:* Debugging YARN AM****
>>
>> ** **
>>
>> Hi all, ****
>>
>> ** **
>>
>> I can successfully debug the MapReduce ApplicationMaster in standalone
>> mode by launching the pi estimator example with this command:****
>>
>> ** **
>>
>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>
>> ** **
>>
>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>> work with the DistributedShell example, presumably because it's not
>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>> learning purposes, how can I debug the DistributedShell example (and other
>> AMs)?****
>>
>> ** **
>>
>> Thanks!****
>>
>> ** **
>>
>> Curtis****
>>
>
>
Re: Debugging YARN AM
Posted by Curtis Ullerich <cu...@gmail.com>.
Update: It looks like I could add the flag at line 515 of
hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
(package org.apache.hadoop.yarn.applications.distributedshell).
I tried this:
vargs.add("-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000");
I recompiled the jar and launched it with
hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar
hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command
whoami
I added a logging statement to make sure the change took effect. It prints,
but it doesn't seem to halt like the MR example (because of the suspend=y
option) even after launching the debugger. Why might this be?
Thanks!
Curtis
On Tue, Jun 18, 2013 at 8:23 AM, Curtis Ullerich <cu...@gmail.com>wrote:
> Hi Devaraj,
>
> That's what I'm not sure how to do. I want to debug by connecting
> remotely, and I'm not sure how to configure that in the code. I haven't
> found anything telling in the docs/source. Can you point me in the right
> direction?
>
> Thanks,
> Curtis
>
>
> On Mon, Jun 17, 2013 at 11:38 PM, Devaraj k <de...@huawei.com> wrote:
>
>> Hi Curtis, ****
>>
>> ** **
>>
>> "yarn.app.mapreduce.am.command-opts" configuration is specific to
>> MRAppMaster. It is not applicable for DistributedShell AM. ****
>>
>> ** **
>>
>> If you want to dump out debug information then you can make use of the
>> debug option of DistributedShell application. If you want to debug by
>> connecting remotely, you need to update the DS application code accordingly.
>> ****
>>
>> ** **
>>
>> Thanks****
>>
>> Devaraj K****
>>
>> ** **
>>
>> *From:* Curtis Ullerich [mailto:curtisullerich@gmail.com]
>> *Sent:* 18 June 2013 08:19
>> *To:* user@hadoop.apache.org
>> *Subject:* Debugging YARN AM****
>>
>> ** **
>>
>> Hi all, ****
>>
>> ** **
>>
>> I can successfully debug the MapReduce ApplicationMaster in standalone
>> mode by launching the pi estimator example with this command:****
>>
>> ** **
>>
>> hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
>> "-Dyarn.app.mapreduce.am.command-opts=-Xdebug
>> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10****
>>
>> ** **
>>
>> and then attaching a debugger to port 8000 using Eclipse. This doesn't
>> work with the DistributedShell example, presumably because it's not
>> configurable with yarn.app.mapreduce.am.command as it's not MapReduce.
>> Looking in yarn-default.xml, I don't see an equivalent parameter. For
>> learning purposes, how can I debug the DistributedShell example (and other
>> AMs)?****
>>
>> ** **
>>
>> Thanks!****
>>
>> ** **
>>
>> Curtis****
>>
>
>
RE: Debugging YARN AM
Posted by Devaraj k <de...@huawei.com>.
Hi Curtis,
"yarn.app.mapreduce.am.command-opts" configuration is specific to MRAppMaster. It is not applicable for DistributedShell AM.
If you want to dump out debug information then you can make use of the debug option of DistributedShell application. If you want to debug by connecting remotely, you need to update the DS application code accordingly.
Thanks
Devaraj K
From: Curtis Ullerich [mailto:curtisullerich@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in standalone mode by launching the pi estimator example with this command:
hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi "-Dyarn.app.mapreduce.am.command-opts=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10
and then attaching a debugger to port 8000 using Eclipse. This doesn't work with the DistributedShell example, presumably because it's not configurable with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in yarn-default.xml, I don't see an equivalent parameter. For learning purposes, how can I debug the DistributedShell example (and other AMs)?
Thanks!
Curtis
RE: Debugging YARN AM
Posted by Devaraj k <de...@huawei.com>.
Hi Curtis,
"yarn.app.mapreduce.am.command-opts" configuration is specific to MRAppMaster. It is not applicable for DistributedShell AM.
If you want to dump out debug information then you can make use of the debug option of DistributedShell application. If you want to debug by connecting remotely, you need to update the DS application code accordingly.
Thanks
Devaraj K
From: Curtis Ullerich [mailto:curtisullerich@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in standalone mode by launching the pi estimator example with this command:
hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi "-Dyarn.app.mapreduce.am.command-opts=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10
and then attaching a debugger to port 8000 using Eclipse. This doesn't work with the DistributedShell example, presumably because it's not configurable with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in yarn-default.xml, I don't see an equivalent parameter. For learning purposes, how can I debug the DistributedShell example (and other AMs)?
Thanks!
Curtis
RE: Debugging YARN AM
Posted by Devaraj k <de...@huawei.com>.
Hi Curtis,
"yarn.app.mapreduce.am.command-opts" configuration is specific to MRAppMaster. It is not applicable for DistributedShell AM.
If you want to dump out debug information then you can make use of the debug option of DistributedShell application. If you want to debug by connecting remotely, you need to update the DS application code accordingly.
Thanks
Devaraj K
From: Curtis Ullerich [mailto:curtisullerich@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in standalone mode by launching the pi estimator example with this command:
hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi "-Dyarn.app.mapreduce.am.command-opts=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10
and then attaching a debugger to port 8000 using Eclipse. This doesn't work with the DistributedShell example, presumably because it's not configurable with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in yarn-default.xml, I don't see an equivalent parameter. For learning purposes, how can I debug the DistributedShell example (and other AMs)?
Thanks!
Curtis
RE: Debugging YARN AM
Posted by Devaraj k <de...@huawei.com>.
Hi Curtis,
"yarn.app.mapreduce.am.command-opts" configuration is specific to MRAppMaster. It is not applicable for DistributedShell AM.
If you want to dump out debug information then you can make use of the debug option of DistributedShell application. If you want to debug by connecting remotely, you need to update the DS application code accordingly.
Thanks
Devaraj K
From: Curtis Ullerich [mailto:curtisullerich@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM
Hi all,
I can successfully debug the MapReduce ApplicationMaster in standalone mode by launching the pi estimator example with this command:
hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi "-Dyarn.app.mapreduce.am.command-opts=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10
and then attaching a debugger to port 8000 using Eclipse. This doesn't work with the DistributedShell example, presumably because it's not configurable with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in yarn-default.xml, I don't see an equivalent parameter. For learning purposes, how can I debug the DistributedShell example (and other AMs)?
Thanks!
Curtis