You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by bo yang <bo...@gmail.com> on 2014/09/02 08:39:08 UTC
Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Hi Guys,
I am thinking how many concurrent jobs a single Resource Manager might be
able to manage? Following is my understanding, please correct me if I am
wrong.
Let's say if we have 1000 concurrent jobs running. Resource Manager will
have 1000 records in memory to manage these jobs. And it will also have
1000 threads, where each thread is waiting for one job to finish.
The memory part will probably be ok. For the 1000 threads, will there be
any potential problem?
Thanks,
Bo
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
Looks great that YARN is mature enough to support so large cluster. It
gives me more confidence to use :)
Thanks a lot!
Best,
Bo
On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> I don't have the exact number about the max concurrent job. FYI, RM is
> multip-threaded, but the threads are working for different purpose. For
> example, scheduler and rmstatestore has their separate thread, and RPC
> calls are on individual threads as well. It's complicated to evaluate the
> upper bound of concurrent apps, but I've heard of the YARN cluster
> deployment on a cluster of thousands of nodes.
>
> Thanks,
> Zhijie
>
>
> On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Zhijie,
>>
>> That is great to know. Thanks!
>>
>> So there seems no be much limit to support large concurrency. To move
>> this question further, what might be the max number of concurrent jobs
>> which one Resource Manager could support? Is there any numbers from your
>> experience?
>>
>> Thanks,
>> Bo
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
>> wrote:
>>
>>> Hi Bo,
>>>
>>> RM doesn't create an individual thread for each running app. The app
>>> life cycle management is event driven. There's a dispatcher, which runs on
>>> one thread to handle the events for all apps.
>>>
>>> Zhijie
>>>
>>>
>>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I am thinking how many concurrent jobs a single Resource Manager might
>>>> be able to manage? Following is my understanding, please correct me if I am
>>>> wrong.
>>>>
>>>> Let's say if we have 1000 concurrent jobs running. Resource Manager
>>>> will have 1000 records in memory to manage these jobs. And it will also
>>>> have 1000 threads, where each thread is waiting for one job to finish.
>>>>
>>>> The memory part will probably be ok. For the 1000 threads, will there
>>>> be any potential problem?
>>>>
>>>> Thanks,
>>>> Bo
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
Looks great that YARN is mature enough to support so large cluster. It
gives me more confidence to use :)
Thanks a lot!
Best,
Bo
On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> I don't have the exact number about the max concurrent job. FYI, RM is
> multip-threaded, but the threads are working for different purpose. For
> example, scheduler and rmstatestore has their separate thread, and RPC
> calls are on individual threads as well. It's complicated to evaluate the
> upper bound of concurrent apps, but I've heard of the YARN cluster
> deployment on a cluster of thousands of nodes.
>
> Thanks,
> Zhijie
>
>
> On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Zhijie,
>>
>> That is great to know. Thanks!
>>
>> So there seems no be much limit to support large concurrency. To move
>> this question further, what might be the max number of concurrent jobs
>> which one Resource Manager could support? Is there any numbers from your
>> experience?
>>
>> Thanks,
>> Bo
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
>> wrote:
>>
>>> Hi Bo,
>>>
>>> RM doesn't create an individual thread for each running app. The app
>>> life cycle management is event driven. There's a dispatcher, which runs on
>>> one thread to handle the events for all apps.
>>>
>>> Zhijie
>>>
>>>
>>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I am thinking how many concurrent jobs a single Resource Manager might
>>>> be able to manage? Following is my understanding, please correct me if I am
>>>> wrong.
>>>>
>>>> Let's say if we have 1000 concurrent jobs running. Resource Manager
>>>> will have 1000 records in memory to manage these jobs. And it will also
>>>> have 1000 threads, where each thread is waiting for one job to finish.
>>>>
>>>> The memory part will probably be ok. For the 1000 threads, will there
>>>> be any potential problem?
>>>>
>>>> Thanks,
>>>> Bo
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
Looks great that YARN is mature enough to support so large cluster. It
gives me more confidence to use :)
Thanks a lot!
Best,
Bo
On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> I don't have the exact number about the max concurrent job. FYI, RM is
> multip-threaded, but the threads are working for different purpose. For
> example, scheduler and rmstatestore has their separate thread, and RPC
> calls are on individual threads as well. It's complicated to evaluate the
> upper bound of concurrent apps, but I've heard of the YARN cluster
> deployment on a cluster of thousands of nodes.
>
> Thanks,
> Zhijie
>
>
> On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Zhijie,
>>
>> That is great to know. Thanks!
>>
>> So there seems no be much limit to support large concurrency. To move
>> this question further, what might be the max number of concurrent jobs
>> which one Resource Manager could support? Is there any numbers from your
>> experience?
>>
>> Thanks,
>> Bo
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
>> wrote:
>>
>>> Hi Bo,
>>>
>>> RM doesn't create an individual thread for each running app. The app
>>> life cycle management is event driven. There's a dispatcher, which runs on
>>> one thread to handle the events for all apps.
>>>
>>> Zhijie
>>>
>>>
>>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I am thinking how many concurrent jobs a single Resource Manager might
>>>> be able to manage? Following is my understanding, please correct me if I am
>>>> wrong.
>>>>
>>>> Let's say if we have 1000 concurrent jobs running. Resource Manager
>>>> will have 1000 records in memory to manage these jobs. And it will also
>>>> have 1000 threads, where each thread is waiting for one job to finish.
>>>>
>>>> The memory part will probably be ok. For the 1000 threads, will there
>>>> be any potential problem?
>>>>
>>>> Thanks,
>>>> Bo
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
Looks great that YARN is mature enough to support so large cluster. It
gives me more confidence to use :)
Thanks a lot!
Best,
Bo
On Tue, Sep 2, 2014 at 2:41 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> I don't have the exact number about the max concurrent job. FYI, RM is
> multip-threaded, but the threads are working for different purpose. For
> example, scheduler and rmstatestore has their separate thread, and RPC
> calls are on individual threads as well. It's complicated to evaluate the
> upper bound of concurrent apps, but I've heard of the YARN cluster
> deployment on a cluster of thousands of nodes.
>
> Thanks,
> Zhijie
>
>
> On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Zhijie,
>>
>> That is great to know. Thanks!
>>
>> So there seems no be much limit to support large concurrency. To move
>> this question further, what might be the max number of concurrent jobs
>> which one Resource Manager could support? Is there any numbers from your
>> experience?
>>
>> Thanks,
>> Bo
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
>> wrote:
>>
>>> Hi Bo,
>>>
>>> RM doesn't create an individual thread for each running app. The app
>>> life cycle management is event driven. There's a dispatcher, which runs on
>>> one thread to handle the events for all apps.
>>>
>>> Zhijie
>>>
>>>
>>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I am thinking how many concurrent jobs a single Resource Manager might
>>>> be able to manage? Following is my understanding, please correct me if I am
>>>> wrong.
>>>>
>>>> Let's say if we have 1000 concurrent jobs running. Resource Manager
>>>> will have 1000 records in memory to manage these jobs. And it will also
>>>> have 1000 threads, where each thread is waiting for one job to finish.
>>>>
>>>> The memory part will probably be ok. For the 1000 threads, will there
>>>> be any potential problem?
>>>>
>>>> Thanks,
>>>> Bo
>>>>
>>>
>>>
>>>
>>> --
>>> Zhijie Shen
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
I don't have the exact number about the max concurrent job. FYI, RM is
multip-threaded, but the threads are working for different purpose. For
example, scheduler and rmstatestore has their separate thread, and RPC
calls are on individual threads as well. It's complicated to evaluate the
upper bound of concurrent apps, but I've heard of the YARN cluster
deployment on a cluster of thousands of nodes.
Thanks,
Zhijie
On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
> Hi Zhijie,
>
> That is great to know. Thanks!
>
> So there seems no be much limit to support large concurrency. To move this
> question further, what might be the max number of concurrent jobs which one
> Resource Manager could support? Is there any numbers from your experience?
>
> Thanks,
> Bo
>
>
>
>
>
>
> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi Bo,
>>
>> RM doesn't create an individual thread for each running app. The app life
>> cycle management is event driven. There's a dispatcher, which runs on one
>> thread to handle the events for all apps.
>>
>> Zhijie
>>
>>
>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> I am thinking how many concurrent jobs a single Resource Manager might
>>> be able to manage? Following is my understanding, please correct me if I am
>>> wrong.
>>>
>>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>>> have 1000 records in memory to manage these jobs. And it will also have
>>> 1000 threads, where each thread is waiting for one job to finish.
>>>
>>> The memory part will probably be ok. For the 1000 threads, will there be
>>> any potential problem?
>>>
>>> Thanks,
>>> Bo
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
I don't have the exact number about the max concurrent job. FYI, RM is
multip-threaded, but the threads are working for different purpose. For
example, scheduler and rmstatestore has their separate thread, and RPC
calls are on individual threads as well. It's complicated to evaluate the
upper bound of concurrent apps, but I've heard of the YARN cluster
deployment on a cluster of thousands of nodes.
Thanks,
Zhijie
On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
> Hi Zhijie,
>
> That is great to know. Thanks!
>
> So there seems no be much limit to support large concurrency. To move this
> question further, what might be the max number of concurrent jobs which one
> Resource Manager could support? Is there any numbers from your experience?
>
> Thanks,
> Bo
>
>
>
>
>
>
> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi Bo,
>>
>> RM doesn't create an individual thread for each running app. The app life
>> cycle management is event driven. There's a dispatcher, which runs on one
>> thread to handle the events for all apps.
>>
>> Zhijie
>>
>>
>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> I am thinking how many concurrent jobs a single Resource Manager might
>>> be able to manage? Following is my understanding, please correct me if I am
>>> wrong.
>>>
>>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>>> have 1000 records in memory to manage these jobs. And it will also have
>>> 1000 threads, where each thread is waiting for one job to finish.
>>>
>>> The memory part will probably be ok. For the 1000 threads, will there be
>>> any potential problem?
>>>
>>> Thanks,
>>> Bo
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
I don't have the exact number about the max concurrent job. FYI, RM is
multip-threaded, but the threads are working for different purpose. For
example, scheduler and rmstatestore has their separate thread, and RPC
calls are on individual threads as well. It's complicated to evaluate the
upper bound of concurrent apps, but I've heard of the YARN cluster
deployment on a cluster of thousands of nodes.
Thanks,
Zhijie
On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
> Hi Zhijie,
>
> That is great to know. Thanks!
>
> So there seems no be much limit to support large concurrency. To move this
> question further, what might be the max number of concurrent jobs which one
> Resource Manager could support? Is there any numbers from your experience?
>
> Thanks,
> Bo
>
>
>
>
>
>
> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi Bo,
>>
>> RM doesn't create an individual thread for each running app. The app life
>> cycle management is event driven. There's a dispatcher, which runs on one
>> thread to handle the events for all apps.
>>
>> Zhijie
>>
>>
>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> I am thinking how many concurrent jobs a single Resource Manager might
>>> be able to manage? Following is my understanding, please correct me if I am
>>> wrong.
>>>
>>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>>> have 1000 records in memory to manage these jobs. And it will also have
>>> 1000 threads, where each thread is waiting for one job to finish.
>>>
>>> The memory part will probably be ok. For the 1000 threads, will there be
>>> any potential problem?
>>>
>>> Thanks,
>>> Bo
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
I don't have the exact number about the max concurrent job. FYI, RM is
multip-threaded, but the threads are working for different purpose. For
example, scheduler and rmstatestore has their separate thread, and RPC
calls are on individual threads as well. It's complicated to evaluate the
upper bound of concurrent apps, but I've heard of the YARN cluster
deployment on a cluster of thousands of nodes.
Thanks,
Zhijie
On Tue, Sep 2, 2014 at 10:42 AM, bo yang <bo...@gmail.com> wrote:
> Hi Zhijie,
>
> That is great to know. Thanks!
>
> So there seems no be much limit to support large concurrency. To move this
> question further, what might be the max number of concurrent jobs which one
> Resource Manager could support? Is there any numbers from your experience?
>
> Thanks,
> Bo
>
>
>
>
>
>
> On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi Bo,
>>
>> RM doesn't create an individual thread for each running app. The app life
>> cycle management is event driven. There's a dispatcher, which runs on one
>> thread to handle the events for all apps.
>>
>> Zhijie
>>
>>
>> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> I am thinking how many concurrent jobs a single Resource Manager might
>>> be able to manage? Following is my understanding, please correct me if I am
>>> wrong.
>>>
>>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>>> have 1000 records in memory to manage these jobs. And it will also have
>>> 1000 threads, where each thread is waiting for one job to finish.
>>>
>>> The memory part will probably be ok. For the 1000 threads, will there be
>>> any potential problem?
>>>
>>> Thanks,
>>> Bo
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
That is great to know. Thanks!
So there seems no be much limit to support large concurrency. To move this
question further, what might be the max number of concurrent jobs which one
Resource Manager could support? Is there any numbers from your experience?
Thanks,
Bo
On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> RM doesn't create an individual thread for each running app. The app life
> cycle management is event driven. There's a dispatcher, which runs on one
> thread to handle the events for all apps.
>
> Zhijie
>
>
> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> I am thinking how many concurrent jobs a single Resource Manager might be
>> able to manage? Following is my understanding, please correct me if I am
>> wrong.
>>
>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>> have 1000 records in memory to manage these jobs. And it will also have
>> 1000 threads, where each thread is waiting for one job to finish.
>>
>> The memory part will probably be ok. For the 1000 threads, will there be
>> any potential problem?
>>
>> Thanks,
>> Bo
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
That is great to know. Thanks!
So there seems no be much limit to support large concurrency. To move this
question further, what might be the max number of concurrent jobs which one
Resource Manager could support? Is there any numbers from your experience?
Thanks,
Bo
On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> RM doesn't create an individual thread for each running app. The app life
> cycle management is event driven. There's a dispatcher, which runs on one
> thread to handle the events for all apps.
>
> Zhijie
>
>
> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> I am thinking how many concurrent jobs a single Resource Manager might be
>> able to manage? Following is my understanding, please correct me if I am
>> wrong.
>>
>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>> have 1000 records in memory to manage these jobs. And it will also have
>> 1000 threads, where each thread is waiting for one job to finish.
>>
>> The memory part will probably be ok. For the 1000 threads, will there be
>> any potential problem?
>>
>> Thanks,
>> Bo
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
That is great to know. Thanks!
So there seems no be much limit to support large concurrency. To move this
question further, what might be the max number of concurrent jobs which one
Resource Manager could support? Is there any numbers from your experience?
Thanks,
Bo
On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> RM doesn't create an individual thread for each running app. The app life
> cycle management is event driven. There's a dispatcher, which runs on one
> thread to handle the events for all apps.
>
> Zhijie
>
>
> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> I am thinking how many concurrent jobs a single Resource Manager might be
>> able to manage? Following is my understanding, please correct me if I am
>> wrong.
>>
>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>> have 1000 records in memory to manage these jobs. And it will also have
>> 1000 threads, where each thread is waiting for one job to finish.
>>
>> The memory part will probably be ok. For the 1000 threads, will there be
>> any potential problem?
>>
>> Thanks,
>> Bo
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by bo yang <bo...@gmail.com>.
Hi Zhijie,
That is great to know. Thanks!
So there seems no be much limit to support large concurrency. To move this
question further, what might be the max number of concurrent jobs which one
Resource Manager could support? Is there any numbers from your experience?
Thanks,
Bo
On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Bo,
>
> RM doesn't create an individual thread for each running app. The app life
> cycle management is event driven. There's a dispatcher, which runs on one
> thread to handle the events for all apps.
>
> Zhijie
>
>
> On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> I am thinking how many concurrent jobs a single Resource Manager might be
>> able to manage? Following is my understanding, please correct me if I am
>> wrong.
>>
>> Let's say if we have 1000 concurrent jobs running. Resource Manager will
>> have 1000 records in memory to manage these jobs. And it will also have
>> 1000 threads, where each thread is waiting for one job to finish.
>>
>> The memory part will probably be ok. For the 1000 threads, will there be
>> any potential problem?
>>
>> Thanks,
>> Bo
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
RM doesn't create an individual thread for each running app. The app life
cycle management is event driven. There's a dispatcher, which runs on one
thread to handle the events for all apps.
Zhijie
On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
> Hi Guys,
>
> I am thinking how many concurrent jobs a single Resource Manager might be
> able to manage? Following is my understanding, please correct me if I am
> wrong.
>
> Let's say if we have 1000 concurrent jobs running. Resource Manager will
> have 1000 records in memory to manage these jobs. And it will also have
> 1000 threads, where each thread is waiting for one job to finish.
>
> The memory part will probably be ok. For the 1000 threads, will there be
> any potential problem?
>
> Thanks,
> Bo
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
RM doesn't create an individual thread for each running app. The app life
cycle management is event driven. There's a dispatcher, which runs on one
thread to handle the events for all apps.
Zhijie
On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
> Hi Guys,
>
> I am thinking how many concurrent jobs a single Resource Manager might be
> able to manage? Following is my understanding, please correct me if I am
> wrong.
>
> Let's say if we have 1000 concurrent jobs running. Resource Manager will
> have 1000 records in memory to manage these jobs. And it will also have
> 1000 threads, where each thread is waiting for one job to finish.
>
> The memory part will probably be ok. For the 1000 threads, will there be
> any potential problem?
>
> Thanks,
> Bo
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
RM doesn't create an individual thread for each running app. The app life
cycle management is event driven. There's a dispatcher, which runs on one
thread to handle the events for all apps.
Zhijie
On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
> Hi Guys,
>
> I am thinking how many concurrent jobs a single Resource Manager might be
> able to manage? Following is my understanding, please correct me if I am
> wrong.
>
> Let's say if we have 1000 concurrent jobs running. Resource Manager will
> have 1000 records in memory to manage these jobs. And it will also have
> 1000 threads, where each thread is waiting for one job to finish.
>
> The memory part will probably be ok. For the 1000 threads, will there be
> any potential problem?
>
> Thanks,
> Bo
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any issue with large concurrency due to single active instance of
YARN Resource Manager?
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Bo,
RM doesn't create an individual thread for each running app. The app life
cycle management is event driven. There's a dispatcher, which runs on one
thread to handle the events for all apps.
Zhijie
On Mon, Sep 1, 2014 at 11:39 PM, bo yang <bo...@gmail.com> wrote:
> Hi Guys,
>
> I am thinking how many concurrent jobs a single Resource Manager might be
> able to manage? Following is my understanding, please correct me if I am
> wrong.
>
> Let's say if we have 1000 concurrent jobs running. Resource Manager will
> have 1000 records in memory to manage these jobs. And it will also have
> 1000 threads, where each thread is waiting for one job to finish.
>
> The memory part will probably be ok. For the 1000 threads, will there be
> any potential problem?
>
> Thanks,
> Bo
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.