Re: What else can be built on top of YARN.

[Rahul Bhattacharjee <ra...@gmail.com>]

Thanks a lot for the responses. I now have a better understanding.

To answer to Jay's question , I think ZK can be used as for coordination
service for a distributed program (you have built it on top of exposed
granular api's) and it doesn't have features like resource management
(including allocation of resources based on requests) of cluster nodes ,
which yarn has.

Rahul


On Thu, May 30, 2013 at 5:59 PM, Jay Vyas <ja...@gmail.com> wrote:

> What is the separation of concerns between YARN and Zookeeper?  That is,
> where does YARN leave off and where does Zookeeper begin?  Or is there some
> overlap....
>
>
> On Thu, May 30, 2013 at 2:42 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Rahul,
>>
>>   It is at least because of the reasons that Vinod listed that makes my
>> life easy for porting my application on to YARN instead of making it work
>> in the Map Reduce framework. The main purpose of me using YARN is to
>> exploit the resource management capabilities of YARN.
>>
>> Thanks,
>> Kishore
>>
>>
>> On Wed, May 29, 2013 at 11:00 PM, Rahul Bhattacharjee <
>> rahul.rec.dgp@gmail.com> wrote:
>>
>>> Thanks for the response Krishna.
>>>
>>> I was wondering if it were possible for using MR to  solve you problem
>>> instead of building the whole stack on top of yarn.
>>> Most likely its not possible , thats why you are building it . I wanted
>>> to know why is that ?
>>>
>>> I am in just trying to find out the need or why we might need to write
>>> the application on yarn.
>>>
>>> Rahul
>>>
>>>
>>> On Wed, May 29, 2013 at 8:23 PM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Rahul,
>>>>
>>>>   I am porting a distributed application that runs on a fixed set of
>>>> given resources to YARN, with the aim of  being able to run it on a
>>>> dynamically selected resources whichever are available at the time of
>>>> running the application.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>>
>>>> On Wed, May 29, 2013 at 8:04 PM, Rahul Bhattacharjee <
>>>> rahul.rec.dgp@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I was going through the motivation behind Yarn. Splitting the
>>>>> responsibility of JT is the major concern.Ultimately the base (Yarn) was
>>>>> built in a generic way for building other generic distributed applications
>>>>> too.
>>>>>
>>>>> I am not able to think of any other parallel processing use case that
>>>>> would be useful to built on top of YARN. I though of a lot of use cases
>>>>> that would be beneficial when run in parallel , but again ,we can do those
>>>>> using map only jobs in MR.
>>>>>
>>>>> Can someone tell me a scenario , where a application can utilize Yarn
>>>>> features or can be built on top of YARN and at the same time , it cannot be
>>>>> done efficiently using MRv2 jobs.
>>>>>
>>>>> thanks,
>>>>> Rahul
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>