You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Thaddeus Diamond <th...@gmail.com> on 2014/09/17 03:46:07 UTC
Swimlanes Usage
Hey guys,
I'm trying to use the swimlanes tool (very nice!) but am getting an error.
I have Tez revision a1dd829 being run and when I run the swimlanes bash
script (added set -x) I get:
[thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
application_1410882226668_0008
+ APPID=application_1410882226668_0008
++ which yarn
+ YARN=/usr/bin/yarn
++ mktemp
+ TMP=/tmp/tmp.kmPS10lCNt
+ echo 'Fetching yarn logs for application_1410882226668_0008'
Fetching yarn logs for application_1410882226668_0008
+ /usr/bin/yarn logs -applicationId application_1410882226668_0008
+ grep HISTORY
+ python swimlane.py -o application_1410882226668_0008.svg
/tmp/tmp.kmPS10lCNt
Traceback (most recent call last):
File "swimlane.py", line 201, in <module>
sys.exit(main(sys.argv[1:]))
File "swimlane.py", line 121, in main
log = AMLog(args[0]).structure()
File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
am.containers = containers
AttributeError: 'NoneType' object has no attribute 'containers'
Thanks,
Thad
RE: Swimlanes Usage
Posted by Bikas Saha <bi...@hortonworks.com>.
That the way YARN does log uploads currently. They are making changes to
that.
Eventually, the swimlanes will be constructed from YARN ATS data and should
be more on demand because ATS data is updated on demand.
*From:* Thaddeus Diamond [mailto:thaddeus.diamond@gmail.com]
*Sent:* Tuesday, September 16, 2014 7:55 PM
*To:* user@tez.apache.org
*Subject:* Re: Swimlanes Usage
You know what, looks like it's because it's a Tez session that is not yet
complete. YARN can't give us application logs until the AM returns. Would
be good in the future to support during-session dumps so I don't have to
restart the whole session. Should I file a JIRA?
On Tue, Sep 16, 2014 at 10:30 PM, Jonathan Eagles <je...@gmail.com> wrote:
Thad, the swimlanes tool may need some added robustness for the case above.
Can you capture the yarn logs for this job to help in debugging?
jeagles
On Tue, Sep 16, 2014 at 8:46 PM, Thaddeus Diamond <
thaddeus.diamond@gmail.com> wrote:
Hey guys,
I'm trying to use the swimlanes tool (very nice!) but am getting an error.
I have Tez revision a1dd829 being run and when I run the swimlanes bash
script (added set -x) I get:
[thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
application_1410882226668_0008
+ APPID=application_1410882226668_0008
++ which yarn
+ YARN=/usr/bin/yarn
++ mktemp
+ TMP=/tmp/tmp.kmPS10lCNt
+ echo 'Fetching yarn logs for application_1410882226668_0008'
Fetching yarn logs for application_1410882226668_0008
+ /usr/bin/yarn logs -applicationId application_1410882226668_0008
+ grep HISTORY
+ python swimlane.py -o application_1410882226668_0008.svg
/tmp/tmp.kmPS10lCNt
Traceback (most recent call last):
File "swimlane.py", line 201, in <module>
sys.exit(main(sys.argv[1:]))
File "swimlane.py", line 121, in main
log = AMLog(args[0]).structure()
File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
am.containers = containers
AttributeError: 'NoneType' object has no attribute 'containers'
Thanks,
Thad
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Swimlanes Usage
Posted by Thaddeus Diamond <th...@gmail.com>.
Also, I know the swimlanes UI has been discussed but I couldn't see an
answer in the mailing lists to what the red lines at the end of container
lifecycles are (I see a bunch that say -100 in bold red print).
On Tue, Sep 16, 2014 at 10:55 PM, Thaddeus Diamond <
thaddeus.diamond@gmail.com> wrote:
> You know what, looks like it's because it's a Tez session that is not yet
> complete. YARN can't give us application logs until the AM returns. Would
> be good in the future to support during-session dumps so I don't have to
> restart the whole session. Should I file a JIRA?
>
> On Tue, Sep 16, 2014 at 10:30 PM, Jonathan Eagles <je...@gmail.com>
> wrote:
>
>> Thad, the swimlanes tool may need some added robustness for the case
>> above. Can you capture the yarn logs for this job to help in debugging?
>>
>> jeagles
>>
>> On Tue, Sep 16, 2014 at 8:46 PM, Thaddeus Diamond <
>> thaddeus.diamond@gmail.com> wrote:
>>
>>> Hey guys,
>>>
>>> I'm trying to use the swimlanes tool (very nice!) but am getting an
>>> error. I have Tez revision a1dd829 being run and when I run the swimlanes
>>> bash script (added set -x) I get:
>>>
>>> [thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
>>> application_1410882226668_0008
>>> + APPID=application_1410882226668_0008
>>> ++ which yarn
>>> + YARN=/usr/bin/yarn
>>> ++ mktemp
>>> + TMP=/tmp/tmp.kmPS10lCNt
>>> + echo 'Fetching yarn logs for application_1410882226668_0008'
>>> Fetching yarn logs for application_1410882226668_0008
>>> + /usr/bin/yarn logs -applicationId application_1410882226668_0008
>>> + grep HISTORY
>>> + python swimlane.py -o application_1410882226668_0008.svg
>>> /tmp/tmp.kmPS10lCNt
>>> Traceback (most recent call last):
>>> File "swimlane.py", line 201, in <module>
>>> sys.exit(main(sys.argv[1:]))
>>> File "swimlane.py", line 121, in main
>>> log = AMLog(args[0]).structure()
>>> File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
>>> am.containers = containers
>>> AttributeError: 'NoneType' object has no attribute 'containers'
>>>
>>> Thanks,
>>> Thad
>>>
>>
>>
>
Re: Swimlanes Usage
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
We are tacking this now in YARN. Some part of it is being tracked at
https://issues.apache.org/jira/browse/YARN-2468 with focus on facilitating
long running applications.
+Vinod
On Tue, Sep 16, 2014 at 7:55 PM, Thaddeus Diamond <
thaddeus.diamond@gmail.com> wrote:
> You know what, looks like it's because it's a Tez session that is not yet
> complete. YARN can't give us application logs until the AM returns. Would
> be good in the future to support during-session dumps so I don't have to
> restart the whole session. Should I file a JIRA?
>
> On Tue, Sep 16, 2014 at 10:30 PM, Jonathan Eagles <je...@gmail.com>
> wrote:
>
>> Thad, the swimlanes tool may need some added robustness for the case
>> above. Can you capture the yarn logs for this job to help in debugging?
>>
>> jeagles
>>
>> On Tue, Sep 16, 2014 at 8:46 PM, Thaddeus Diamond <
>> thaddeus.diamond@gmail.com> wrote:
>>
>>> Hey guys,
>>>
>>> I'm trying to use the swimlanes tool (very nice!) but am getting an
>>> error. I have Tez revision a1dd829 being run and when I run the swimlanes
>>> bash script (added set -x) I get:
>>>
>>> [thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
>>> application_1410882226668_0008
>>> + APPID=application_1410882226668_0008
>>> ++ which yarn
>>> + YARN=/usr/bin/yarn
>>> ++ mktemp
>>> + TMP=/tmp/tmp.kmPS10lCNt
>>> + echo 'Fetching yarn logs for application_1410882226668_0008'
>>> Fetching yarn logs for application_1410882226668_0008
>>> + /usr/bin/yarn logs -applicationId application_1410882226668_0008
>>> + grep HISTORY
>>> + python swimlane.py -o application_1410882226668_0008.svg
>>> /tmp/tmp.kmPS10lCNt
>>> Traceback (most recent call last):
>>> File "swimlane.py", line 201, in <module>
>>> sys.exit(main(sys.argv[1:]))
>>> File "swimlane.py", line 121, in main
>>> log = AMLog(args[0]).structure()
>>> File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
>>> am.containers = containers
>>> AttributeError: 'NoneType' object has no attribute 'containers'
>>>
>>> Thanks,
>>> Thad
>>>
>>
>>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Swimlanes Usage
Posted by Thaddeus Diamond <th...@gmail.com>.
You know what, looks like it's because it's a Tez session that is not yet
complete. YARN can't give us application logs until the AM returns. Would
be good in the future to support during-session dumps so I don't have to
restart the whole session. Should I file a JIRA?
On Tue, Sep 16, 2014 at 10:30 PM, Jonathan Eagles <je...@gmail.com> wrote:
> Thad, the swimlanes tool may need some added robustness for the case
> above. Can you capture the yarn logs for this job to help in debugging?
>
> jeagles
>
> On Tue, Sep 16, 2014 at 8:46 PM, Thaddeus Diamond <
> thaddeus.diamond@gmail.com> wrote:
>
>> Hey guys,
>>
>> I'm trying to use the swimlanes tool (very nice!) but am getting an
>> error. I have Tez revision a1dd829 being run and when I run the swimlanes
>> bash script (added set -x) I get:
>>
>> [thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
>> application_1410882226668_0008
>> + APPID=application_1410882226668_0008
>> ++ which yarn
>> + YARN=/usr/bin/yarn
>> ++ mktemp
>> + TMP=/tmp/tmp.kmPS10lCNt
>> + echo 'Fetching yarn logs for application_1410882226668_0008'
>> Fetching yarn logs for application_1410882226668_0008
>> + /usr/bin/yarn logs -applicationId application_1410882226668_0008
>> + grep HISTORY
>> + python swimlane.py -o application_1410882226668_0008.svg
>> /tmp/tmp.kmPS10lCNt
>> Traceback (most recent call last):
>> File "swimlane.py", line 201, in <module>
>> sys.exit(main(sys.argv[1:]))
>> File "swimlane.py", line 121, in main
>> log = AMLog(args[0]).structure()
>> File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
>> am.containers = containers
>> AttributeError: 'NoneType' object has no attribute 'containers'
>>
>> Thanks,
>> Thad
>>
>
>
Re: Swimlanes Usage
Posted by Jonathan Eagles <je...@gmail.com>.
Thad, the swimlanes tool may need some added robustness for the case above.
Can you capture the yarn logs for this job to help in debugging?
jeagles
On Tue, Sep 16, 2014 at 8:46 PM, Thaddeus Diamond <
thaddeus.diamond@gmail.com> wrote:
> Hey guys,
>
> I'm trying to use the swimlanes tool (very nice!) but am getting an error.
> I have Tez revision a1dd829 being run and when I run the swimlanes bash
> script (added set -x) I get:
>
> [thaddeusdiamond@hadapt ~]$ ./yarn-swimlanes.sh
> application_1410882226668_0008
> + APPID=application_1410882226668_0008
> ++ which yarn
> + YARN=/usr/bin/yarn
> ++ mktemp
> + TMP=/tmp/tmp.kmPS10lCNt
> + echo 'Fetching yarn logs for application_1410882226668_0008'
> Fetching yarn logs for application_1410882226668_0008
> + /usr/bin/yarn logs -applicationId application_1410882226668_0008
> + grep HISTORY
> + python swimlane.py -o application_1410882226668_0008.svg
> /tmp/tmp.kmPS10lCNt
> Traceback (most recent call last):
> File "swimlane.py", line 201, in <module>
> sys.exit(main(sys.argv[1:]))
> File "swimlane.py", line 121, in main
> log = AMLog(args[0]).structure()
> File "/home/thaddeusdiamond/amlogparser.py", line 203, in structure
> am.containers = containers
> AttributeError: 'NoneType' object has no attribute 'containers'
>
> Thanks,
> Thad
>