You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pradeep Gollakota <pr...@gmail.com> on 2013/09/20 20:24:38 UTC
Help writing a YARN application
Hi All,
I've been trying to write a Yarn application and I'm completely lost. I'm
using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
sample code to github at https://github.com/pradeepg26/sample-yarn
The problem is that my application master is exiting with a status of 1
(I'm expecting that since my code isn't complete yet). But I have no logs
that I can examine. So I'm not sure if the error I'm getting is the error
I'm expecting. I've attached the nodemanger and resourcemanager logs for
your reference as well.
How can I get started on writing YARN applications beyond the initial
tutorial?
Thanks for any help/pointers!
Pradeep
Re: Help writing a YARN application
Posted by Personal <ni...@gmail.com>.
Hi Pradeep,
You can also look at Weave - A simpler thread abstraction over YARN.
http://github.com/continuuity/weave
Runs on multiple HDs.
Nitin
On Mon, Sep 23, 2013 at 10:20 PM, Pradeep Gollakota <pr...@gmail.com>
wrote:
> Hi Arun,
> Thanks so much for your reply. I looked at the app you linked, it's way
> easier to understand. Unfortunately, I can't use YarnClient class. It
> doesn't seem to be in the 2.0.0 branch. Our production environment is using
> cdh-4.x and it looks like they're tracking the 2.0.x branch.
> However, I have looked at other implementations (notably Kitten and Giraph)
> for inspiration and able to get my containers successfully launching.
> Thanks for the help!
> - Pradeep
> On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> I looked at your code, it's using really old apis/protocols which are
>> significantly different now. See hadoop-2.1.0-beta release for latest
>> apis/protocols.
>>
>> Also, you should really be using yarn client module rather than raw
>> protocols.
>>
>> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>>
>> thanks,
>> Arun
>>
>> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I've been trying to write a Yarn application and I'm completely lost. I'm
>> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
>> sample code to github at https://github.com/pradeepg26/sample-yarn
>>
>> The problem is that my application master is exiting with a status of 1
>> (I'm expecting that since my code isn't complete yet). But I have no logs
>> that I can examine. So I'm not sure if the error I'm getting is the error
>> I'm expecting. I've attached the nodemanger and resourcemanager logs for
>> your reference as well.
>>
>> How can I get started on writing YARN applications beyond the initial
>> tutorial?
>>
>> Thanks for any help/pointers!
>>
>> Pradeep
>> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
>> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Personal <ni...@gmail.com>.
Hi Pradeep,
You can also look at Weave - A simpler thread abstraction over YARN.
http://github.com/continuuity/weave
Runs on multiple HDs.
Nitin
On Mon, Sep 23, 2013 at 10:20 PM, Pradeep Gollakota <pr...@gmail.com>
wrote:
> Hi Arun,
> Thanks so much for your reply. I looked at the app you linked, it's way
> easier to understand. Unfortunately, I can't use YarnClient class. It
> doesn't seem to be in the 2.0.0 branch. Our production environment is using
> cdh-4.x and it looks like they're tracking the 2.0.x branch.
> However, I have looked at other implementations (notably Kitten and Giraph)
> for inspiration and able to get my containers successfully launching.
> Thanks for the help!
> - Pradeep
> On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> I looked at your code, it's using really old apis/protocols which are
>> significantly different now. See hadoop-2.1.0-beta release for latest
>> apis/protocols.
>>
>> Also, you should really be using yarn client module rather than raw
>> protocols.
>>
>> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>>
>> thanks,
>> Arun
>>
>> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I've been trying to write a Yarn application and I'm completely lost. I'm
>> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
>> sample code to github at https://github.com/pradeepg26/sample-yarn
>>
>> The problem is that my application master is exiting with a status of 1
>> (I'm expecting that since my code isn't complete yet). But I have no logs
>> that I can examine. So I'm not sure if the error I'm getting is the error
>> I'm expecting. I've attached the nodemanger and resourcemanager logs for
>> your reference as well.
>>
>> How can I get started on writing YARN applications beyond the initial
>> tutorial?
>>
>> Thanks for any help/pointers!
>>
>> Pradeep
>> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
>> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Personal <ni...@gmail.com>.
Hi Pradeep,
You can also look at Weave - A simpler thread abstraction over YARN.
http://github.com/continuuity/weave
Runs on multiple HDs.
Nitin
On Mon, Sep 23, 2013 at 10:20 PM, Pradeep Gollakota <pr...@gmail.com>
wrote:
> Hi Arun,
> Thanks so much for your reply. I looked at the app you linked, it's way
> easier to understand. Unfortunately, I can't use YarnClient class. It
> doesn't seem to be in the 2.0.0 branch. Our production environment is using
> cdh-4.x and it looks like they're tracking the 2.0.x branch.
> However, I have looked at other implementations (notably Kitten and Giraph)
> for inspiration and able to get my containers successfully launching.
> Thanks for the help!
> - Pradeep
> On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> I looked at your code, it's using really old apis/protocols which are
>> significantly different now. See hadoop-2.1.0-beta release for latest
>> apis/protocols.
>>
>> Also, you should really be using yarn client module rather than raw
>> protocols.
>>
>> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>>
>> thanks,
>> Arun
>>
>> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I've been trying to write a Yarn application and I'm completely lost. I'm
>> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
>> sample code to github at https://github.com/pradeepg26/sample-yarn
>>
>> The problem is that my application master is exiting with a status of 1
>> (I'm expecting that since my code isn't complete yet). But I have no logs
>> that I can examine. So I'm not sure if the error I'm getting is the error
>> I'm expecting. I've attached the nodemanger and resourcemanager logs for
>> your reference as well.
>>
>> How can I get started on writing YARN applications beyond the initial
>> tutorial?
>>
>> Thanks for any help/pointers!
>>
>> Pradeep
>> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
>> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Personal <ni...@gmail.com>.
Hi Pradeep,
You can also look at Weave - A simpler thread abstraction over YARN.
http://github.com/continuuity/weave
Runs on multiple HDs.
Nitin
On Mon, Sep 23, 2013 at 10:20 PM, Pradeep Gollakota <pr...@gmail.com>
wrote:
> Hi Arun,
> Thanks so much for your reply. I looked at the app you linked, it's way
> easier to understand. Unfortunately, I can't use YarnClient class. It
> doesn't seem to be in the 2.0.0 branch. Our production environment is using
> cdh-4.x and it looks like they're tracking the 2.0.x branch.
> However, I have looked at other implementations (notably Kitten and Giraph)
> for inspiration and able to get my containers successfully launching.
> Thanks for the help!
> - Pradeep
> On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> I looked at your code, it's using really old apis/protocols which are
>> significantly different now. See hadoop-2.1.0-beta release for latest
>> apis/protocols.
>>
>> Also, you should really be using yarn client module rather than raw
>> protocols.
>>
>> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>>
>> thanks,
>> Arun
>>
>> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> I've been trying to write a Yarn application and I'm completely lost. I'm
>> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
>> sample code to github at https://github.com/pradeepg26/sample-yarn
>>
>> The problem is that my application master is exiting with a status of 1
>> (I'm expecting that since my code isn't complete yet). But I have no logs
>> that I can examine. So I'm not sure if the error I'm getting is the error
>> I'm expecting. I've attached the nodemanger and resourcemanager logs for
>> your reference as well.
>>
>> How can I get started on writing YARN applications beyond the initial
>> tutorial?
>>
>> Thanks for any help/pointers!
>>
>> Pradeep
>> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
>> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Pradeep Gollakota <pr...@gmail.com>.
Hi Arun,
Thanks so much for your reply. I looked at the app you linked, it's way
easier to understand. Unfortunately, I can't use YarnClient class. It
doesn't seem to be in the 2.0.0 branch. Our production environment is using
cdh-4.x and it looks like they're tracking the 2.0.x branch.
However, I have looked at other implementations (notably Kitten and Giraph)
for inspiration and able to get my containers successfully launching.
Thanks for the help!
- Pradeep
On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I looked at your code, it's using really old apis/protocols which are
> significantly different now. See hadoop-2.1.0-beta release for latest
> apis/protocols.
>
> Also, you should really be using yarn client module rather than raw
> protocols.
>
> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>
> thanks,
> Arun
>
> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
> wrote:
>
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Pradeep Gollakota <pr...@gmail.com>.
Hi Arun,
Thanks so much for your reply. I looked at the app you linked, it's way
easier to understand. Unfortunately, I can't use YarnClient class. It
doesn't seem to be in the 2.0.0 branch. Our production environment is using
cdh-4.x and it looks like they're tracking the 2.0.x branch.
However, I have looked at other implementations (notably Kitten and Giraph)
for inspiration and able to get my containers successfully launching.
Thanks for the help!
- Pradeep
On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I looked at your code, it's using really old apis/protocols which are
> significantly different now. See hadoop-2.1.0-beta release for latest
> apis/protocols.
>
> Also, you should really be using yarn client module rather than raw
> protocols.
>
> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>
> thanks,
> Arun
>
> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
> wrote:
>
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Pradeep Gollakota <pr...@gmail.com>.
Hi Arun,
Thanks so much for your reply. I looked at the app you linked, it's way
easier to understand. Unfortunately, I can't use YarnClient class. It
doesn't seem to be in the 2.0.0 branch. Our production environment is using
cdh-4.x and it looks like they're tracking the 2.0.x branch.
However, I have looked at other implementations (notably Kitten and Giraph)
for inspiration and able to get my containers successfully launching.
Thanks for the help!
- Pradeep
On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I looked at your code, it's using really old apis/protocols which are
> significantly different now. See hadoop-2.1.0-beta release for latest
> apis/protocols.
>
> Also, you should really be using yarn client module rather than raw
> protocols.
>
> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>
> thanks,
> Arun
>
> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
> wrote:
>
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Pradeep Gollakota <pr...@gmail.com>.
Hi Arun,
Thanks so much for your reply. I looked at the app you linked, it's way
easier to understand. Unfortunately, I can't use YarnClient class. It
doesn't seem to be in the 2.0.0 branch. Our production environment is using
cdh-4.x and it looks like they're tracking the 2.0.x branch.
However, I have looked at other implementations (notably Kitten and Giraph)
for inspiration and able to get my containers successfully launching.
Thanks for the help!
- Pradeep
On Mon, Sep 23, 2013 at 4:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I looked at your code, it's using really old apis/protocols which are
> significantly different now. See hadoop-2.1.0-beta release for latest
> apis/protocols.
>
> Also, you should really be using yarn client module rather than raw
> protocols.
>
> See https://github.com/hortonworks/simple-yarn-app for a simple example.
>
> thanks,
> Arun
>
> On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com>
> wrote:
>
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log>
> <yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Arun C Murthy <ac...@hortonworks.com>.
I looked at your code, it's using really old apis/protocols which are significantly different now. See hadoop-2.1.0-beta release for latest apis/protocols.
Also, you should really be using yarn client module rather than raw protocols.
See https://github.com/hortonworks/simple-yarn-app for a simple example.
thanks,
Arun
On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com> wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1 (I'm expecting that since my code isn't complete yet). But I have no logs that I can examine. So I'm not sure if the error I'm getting is the error I'm expecting. I've attached the nodemanger and resourcemanager logs for your reference as well.
>
> How can I get started on writing YARN applications beyond the initial tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log><yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I tried something , see if this helps. Its incomplete though.
https://github.com/brahul/singular
Thanks,
Rahul
On Fri, Sep 20, 2013 at 11:54 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
>
Re: Help writing a YARN application
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I tried something , see if this helps. Its incomplete though.
https://github.com/brahul/singular
Thanks,
Rahul
On Fri, Sep 20, 2013 at 11:54 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
>
Re: Help writing a YARN application
Posted by Arun C Murthy <ac...@hortonworks.com>.
I looked at your code, it's using really old apis/protocols which are significantly different now. See hadoop-2.1.0-beta release for latest apis/protocols.
Also, you should really be using yarn client module rather than raw protocols.
See https://github.com/hortonworks/simple-yarn-app for a simple example.
thanks,
Arun
On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com> wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1 (I'm expecting that since my code isn't complete yet). But I have no logs that I can examine. So I'm not sure if the error I'm getting is the error I'm expecting. I've attached the nodemanger and resourcemanager logs for your reference as well.
>
> How can I get started on writing YARN applications beyond the initial tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log><yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Arun C Murthy <ac...@hortonworks.com>.
I looked at your code, it's using really old apis/protocols which are significantly different now. See hadoop-2.1.0-beta release for latest apis/protocols.
Also, you should really be using yarn client module rather than raw protocols.
See https://github.com/hortonworks/simple-yarn-app for a simple example.
thanks,
Arun
On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com> wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1 (I'm expecting that since my code isn't complete yet). But I have no logs that I can examine. So I'm not sure if the error I'm getting is the error I'm expecting. I've attached the nodemanger and resourcemanager logs for your reference as well.
>
> How can I get started on writing YARN applications beyond the initial tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log><yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Help writing a YARN application
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I tried something , see if this helps. Its incomplete though.
https://github.com/brahul/singular
Thanks,
Rahul
On Fri, Sep 20, 2013 at 11:54 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
>
Re: Help writing a YARN application
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I tried something , see if this helps. Its incomplete though.
https://github.com/brahul/singular
Thanks,
Rahul
On Fri, Sep 20, 2013 at 11:54 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm
> using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my
> sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1
> (I'm expecting that since my code isn't complete yet). But I have no logs
> that I can examine. So I'm not sure if the error I'm getting is the error
> I'm expecting. I've attached the nodemanger and resourcemanager logs for
> your reference as well.
>
> How can I get started on writing YARN applications beyond the initial
> tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
>
Re: Help writing a YARN application
Posted by Arun C Murthy <ac...@hortonworks.com>.
I looked at your code, it's using really old apis/protocols which are significantly different now. See hadoop-2.1.0-beta release for latest apis/protocols.
Also, you should really be using yarn client module rather than raw protocols.
See https://github.com/hortonworks/simple-yarn-app for a simple example.
thanks,
Arun
On Sep 20, 2013, at 11:24 AM, Pradeep Gollakota <pr...@gmail.com> wrote:
> Hi All,
>
> I've been trying to write a Yarn application and I'm completely lost. I'm using Hadoop 2.0.0-cdh4.4.0 (Cloudera distribution). I've uploaded my sample code to github at https://github.com/pradeepg26/sample-yarn
>
> The problem is that my application master is exiting with a status of 1 (I'm expecting that since my code isn't complete yet). But I have no logs that I can examine. So I'm not sure if the error I'm getting is the error I'm expecting. I've attached the nodemanger and resourcemanager logs for your reference as well.
>
> How can I get started on writing YARN applications beyond the initial tutorial?
>
> Thanks for any help/pointers!
>
> Pradeep
> <yarn-yarn-nodemanager-pradeep-gollakota.vm.lithium.com.log><yarn-yarn-resourcemanager-pradeep-gollakota.vm.lithium.com.log>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.