resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Hi, all

In HDInsight, we (Microsoft) use Livy as the Spark job submission service.
We keep seeing the customers fall into the problem when they submit many
concurrent applications to the system, or recover livy from a state with
many concurrent applications

By looking at the code and the customers' exception stack, we lock down the
problem to the application monitoring module where a new thread is created
for each application.

To resolve the issue, we propose a actor-based design of application
monitoring module and share it here (as new JIRA seems not working
yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
<https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*

We are glad to hear feedbacks from the community and improve the design
before we start implementing it!

Best,

Nan

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

PDF version

On Tue, Aug 15, 2017 at 2:22 PM, Nan Zhu <zh...@gmail.com> wrote:

> I also attached the discarded version of design here
>
> Best,
>
> Nan
>
> On Tue, Aug 15, 2017 at 2:20 PM, Nan Zhu <zh...@gmail.com> wrote:
>
>> Hi, Marcelo,
>>
>> Yes, essentially it is using multiple threads talking with YARN.
>>
>> The key design consideration here is that how you model the state of
>> applications, if in actor, then there will be no synchronization involved
>> and yielding a cleaner design; if in a shared data structure, you will have
>> to be careful about coordinating threads here (we actually have a design
>> based on shared data structure and we eventually discard to pursue a
>> cleaner one).
>>
>> I think bulk API can make life easier comparing to the shared data
>> structure, but it raises up two questions
>>
>> 1. Are we going to update all applications in the uniform pace, even they
>> are submitted in different time?
>>
>> 2. Are we going to use a single thread for everything, including
>> send/recv req/res and parse, etc.
>>
>> and we still need to deal with some synchronization,
>>
>> What do you think?
>>
>> Best,
>>
>> Nan
>>
>>
>>
>>
>>
>>
>> On Tue, Aug 15, 2017 at 11:53 AM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>>
>>> Hmm, I remember this... it was left as a "todo" item when the app
>>> monitoring was added.
>>>
>>> The document you wrote seems to be a long way of saying you'll have a
>>> few threads talking to YARN and updating the state of application
>>> handles in Livy. Is that right?
>>>
>>> I would investigate whether there's any API in YARN to do a bulk get
>>> of running applications with a particular filter; then you could make
>>> a single call to YARN periodically to get the state of all apps that
>>> Livy started.
>>>
>>>
>>> On Mon, Aug 14, 2017 at 2:35 PM, Nan Zhu <zh...@gmail.com> wrote:
>>> > Hi, all
>>> >
>>> > In HDInsight, we (Microsoft) use Livy as the Spark job submission
>>> service.
>>> > We keep seeing the customers fall into the problem when they submit
>>> many
>>> > concurrent applications to the system, or recover livy from a state
>>> with
>>> > many concurrent applications
>>> >
>>> > By looking at the code and the customers' exception stack, we lock
>>> down the
>>> > problem to the application monitoring module where a new thread is
>>> created
>>> > for each application.
>>> >
>>> > To resolve the issue, we propose a actor-based design of application
>>> > monitoring module and share it here (as new JIRA seems not working
>>> > yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P
>>> -nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
>>> > <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P
>>> -nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*
>>> >
>>> > We are glad to hear feedbacks from the community and improve the design
>>> > before we start implementing it!
>>> >
>>> > Best,
>>> >
>>> > Nan
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>
>>
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

I also attached the discarded version of design here

Best,

Nan

On Tue, Aug 15, 2017 at 2:20 PM, Nan Zhu <zh...@gmail.com> wrote:

> Hi, Marcelo,
>
> Yes, essentially it is using multiple threads talking with YARN.
>
> The key design consideration here is that how you model the state of
> applications, if in actor, then there will be no synchronization involved
> and yielding a cleaner design; if in a shared data structure, you will have
> to be careful about coordinating threads here (we actually have a design
> based on shared data structure and we eventually discard to pursue a
> cleaner one).
>
> I think bulk API can make life easier comparing to the shared data
> structure, but it raises up two questions
>
> 1. Are we going to update all applications in the uniform pace, even they
> are submitted in different time?
>
> 2. Are we going to use a single thread for everything, including send/recv
> req/res and parse, etc.
>
> and we still need to deal with some synchronization,
>
> What do you think?
>
> Best,
>
> Nan
>
>
>
>
>
>
> On Tue, Aug 15, 2017 at 11:53 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> Hmm, I remember this... it was left as a "todo" item when the app
>> monitoring was added.
>>
>> The document you wrote seems to be a long way of saying you'll have a
>> few threads talking to YARN and updating the state of application
>> handles in Livy. Is that right?
>>
>> I would investigate whether there's any API in YARN to do a bulk get
>> of running applications with a particular filter; then you could make
>> a single call to YARN periodically to get the state of all apps that
>> Livy started.
>>
>>
>> On Mon, Aug 14, 2017 at 2:35 PM, Nan Zhu <zh...@gmail.com> wrote:
>> > Hi, all
>> >
>> > In HDInsight, we (Microsoft) use Livy as the Spark job submission
>> service.
>> > We keep seeing the customers fall into the problem when they submit many
>> > concurrent applications to the system, or recover livy from a state with
>> > many concurrent applications
>> >
>> > By looking at the code and the customers' exception stack, we lock down
>> the
>> > problem to the application monitoring module where a new thread is
>> created
>> > for each application.
>> >
>> > To resolve the issue, we propose a actor-based design of application
>> > monitoring module and share it here (as new JIRA seems not working
>> > yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P
>> -nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
>> > <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P
>> -nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*
>> >
>> > We are glad to hear feedbacks from the community and improve the design
>> > before we start implementing it!
>> >
>> > Best,
>> >
>> > Nan
>>
>>
>>
>> --
>> Marcelo
>>
>
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Thanks for the reply, Meisam

Looks good to meet our current scenarios

Rest APIs looks like more powerful but it needs to replace the current
YarnClient with a self-made RestClient



On Wed, Aug 16, 2017 at 2:23 PM, Meisam Fathi <me...@gmail.com>
wrote:

> Hi Nan,
>
> In the highlighted line
> >
> > https://github.com/apache/incubator-livy/pull/36/files#diff-
> a3f879755cfe10a678cc08ddbe60a4d3R75
> >
> > I assume that it will get the reports of all applications in YARN, even
> > they are finished?
>
>
> That's right. That line will return reports for all Spark Applications,
> even applications that completed a long time ago. For us YARN retains
> reports for a few thousand completed applications (not a big concern).
>
> Livy needs to get the reports for applications that finished recently, but
> I didn't find an API in YARN 2.7 to get those only reports.
>
> Thanks,
> Meisam
>
> >
> >
>
> > On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <me...@gmail.com>
> > wrote:
> >
> > > Hi Nan,
> > >
> > >
> > > >
> > > > my question related to the undergoing discussion is simply "have you
> > seen
> > > > any performance issue in
> > > >
> > > > https://github.com/apache/incubator-livy/pull/36/files#diff-
> > > a3f879755cfe10a678cc08ddbe60a4d3R75
> > > > ?
> > > > <https://github.com/apache/incubator-livy/pull/36/files#diff-
> > > a3f879755cfe10a678cc08ddbe60a4d3R75?>
> > > > "
> > > >
> > > > The short answer is yes. This PR fixes one part of the scalability
> > > problem, which is, it prevents Livy from creating many
> > > yarnAppMinotorThreads. But the two other parts are still there
> > >
> > > 1. one call to spark-submit for each application
> > > 2. once thread that waits for the exit code of spark-submit.
> > >
> > > Out of these two problems, calling one spark-submit per application is
> > the
> > > biggest problem, but it can be solved by adding more Livy servers. We
> > > modified Livy so if an application status changes on one Livy instance,
> > all
> > > other Livy instances get the updated information about the application.
> > > From users' perspective, this is transparent because users just see the
> > > load balancer.
> > >
> > > So, refactoring the yarn poll mechanism + a load balancer and a grid of
> > > Livy servers fixed the scalability issue.
> > >
> > > On the performance of the code itself, we have not had an issue. The
> time
> > > consuming parts in the code are calls to YARN and not filtering and
> > > updating the data structures. On memory usage, this all needs less than
> > 1GB
> > > at peak time.
> > >
> > > I hope this answers your question.
> > >
> > > Thanks,
> > > Meisam
> > >
> > >
> > > > We have several scenarios that a large volume of applications are
> > > submitted
> > > > to YARN every day and it easily accumulates a lot to be fetched with
> > this
> > > > call
> > > >
> > > > Best,
> > > >
> > > > Nan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <
> meisam.fathi@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Here are my two pennies on both designs (actor-based design vs.
> > > > > single-thread polling design)
> > > > >
> > > > > *Single-thread polling design*
> > > > > We implemented a single-thread polling mechanism for Yarn here at
> > > PayPal.
> > > > > Our solution is more involved because we added many new features to
> > > Livy
> > > > > that we had to consider when we refactored Livy's YARN interface.
> But
> > > we
> > > > > are willing to hammer our changes so it suits the need of the Livy
> > > > > community best :-)
> > > > >
> > > > > *Actor-based design*
> > > > > It seems to me that the proposed actor based design (
> > > > > https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> > > > > nbTQTdDFXl2XQhXDiwA/edit)
> > > > > needs a few more messages and actors. Here is why.
> > > > > Livy makes three (blocking) calls to YARN
> > > > > 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> > > > > 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which
> > > gives
> > > > > Livy `getAMContainerId`
> > > > > 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
> > > > >
> > > > > The result of the previous call is needed to make the next call.
> The
> > > > > proposed actor system needs to be designed to handles all these
> > > blocking
> > > > > calls.
> > > > >
> > > > > I do agree that actor based design is cleaner and more
> maintainable.
> > > But
> > > > we
> > > > > had to discard it because it adds more dependencies to Livy. We
> faced
> > > too
> > > > > many dependency-version-mismatch problems with Livy interactive
> > > sessions
> > > > > (when applications depend on a different version of a library that
> is
> > > > used
> > > > > internally by Livy). If the livy community prefers an actor based
> > > design,
> > > > > we are willing to reimplement our changes with an actor system.
> > > > >
> > > > > Finally, either design is only the first step in fixing this
> > particular
> > > > > scalability problem. The reason is that the *"yarnAppMinotorThread"
> > is
> > > > not
> > > > > the only thread that Livy spawns per Spark application.* For batch
> > > jobs,
> > > > > Livy
> > > > > 1. calls spark-submit, which lunches a new JVM (an operations that
> is
> > > far
> > > > > more heavy than creating a thread and can easily drain the system)
> > > > > 2. It create a thread that waits for the exist code of
> spark-submit.
> > > Even
> > > > > though this thread is "short-lived", at peak time thousands of such
> > > > threads
> > > > > are created in a few seconds.
> > > > >
> > > > > I created a PR with our modifications to it.
> > > > >
> > > > > https://github.com/apache/incubator-livy/pull/36
> > > > >
> > > > > Thanks,
> > > > > Meisam
> > > > >
> > > > > On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <
> vanzin@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> > > > > > <ar...@microsoft.com.invalid> wrote:
> > > > > > > 1. Additional copy of states in Livy which can be queried from
> > YARN
> > > > on
> > > > > > request.
> > > > > >
> > > > > > Not sure I follow.
> > > > > >
> > > > > > > 2. The design is not event driven and may waste querying YARN
> > > > > > unnecessarily when no actual user/external request is pending.
> > > > > >
> > > > > > You don't need to keep querying YARN if there are no apps to
> > monitor.
> > > > > >
> > > > > > > 3. There will always be an issue with stale data and update
> > latency
> > > > > > between actual YARN state and Livy state map.
> > > > > >
> > > > > > That is also the case with a thread pool that has less threads
> than
> > > > > > the number of apps being monitored, if making one request per
> app.
> > > > > >
> > > > > > > 4. Size and latency of the response in bulk querying YARN is
> > > unknown.
> > > > > >
> > > > > > That is also unknown when making multiple requests from multiple
> > > > > > threads, unless you investigate the internal implementation of
> both
> > > > > > YARN clients and servers.
> > > > > >
> > > > > > > 5. YARN bulk API needs to support filtering at the query level.
> > > > > >
> > > > > > Yes, I mentioned that in my original response and really was just
> > > > > > expecting Nan to do a quick investigation of that implementation
> > > > > > option.
> > > > > >
> > > > > > He finally did and it seems that the API only exists through the
> > REST
> > > > > > interface, so this all is moot.
> > > > > >
> > > > > > --
> > > > > > Marcelo
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Hi Nan,

In the highlighted line
>
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75
>
> I assume that it will get the reports of all applications in YARN, even
> they are finished?


That's right. That line will return reports for all Spark Applications,
even applications that completed a long time ago. For us YARN retains
reports for a few thousand completed applications (not a big concern).

Livy needs to get the reports for applications that finished recently, but
I didn't find an API in YARN 2.7 to get those only reports.

Thanks,
Meisam

>
>

> On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <me...@gmail.com>
> wrote:
>
> > Hi Nan,
> >
> >
> > >
> > > my question related to the undergoing discussion is simply "have you
> seen
> > > any performance issue in
> > >
> > > https://github.com/apache/incubator-livy/pull/36/files#diff-
> > a3f879755cfe10a678cc08ddbe60a4d3R75
> > > ?
> > > <https://github.com/apache/incubator-livy/pull/36/files#diff-
> > a3f879755cfe10a678cc08ddbe60a4d3R75?>
> > > "
> > >
> > > The short answer is yes. This PR fixes one part of the scalability
> > problem, which is, it prevents Livy from creating many
> > yarnAppMinotorThreads. But the two other parts are still there
> >
> > 1. one call to spark-submit for each application
> > 2. once thread that waits for the exit code of spark-submit.
> >
> > Out of these two problems, calling one spark-submit per application is
> the
> > biggest problem, but it can be solved by adding more Livy servers. We
> > modified Livy so if an application status changes on one Livy instance,
> all
> > other Livy instances get the updated information about the application.
> > From users' perspective, this is transparent because users just see the
> > load balancer.
> >
> > So, refactoring the yarn poll mechanism + a load balancer and a grid of
> > Livy servers fixed the scalability issue.
> >
> > On the performance of the code itself, we have not had an issue. The time
> > consuming parts in the code are calls to YARN and not filtering and
> > updating the data structures. On memory usage, this all needs less than
> 1GB
> > at peak time.
> >
> > I hope this answers your question.
> >
> > Thanks,
> > Meisam
> >
> >
> > > We have several scenarios that a large volume of applications are
> > submitted
> > > to YARN every day and it easily accumulates a lot to be fetched with
> this
> > > call
> > >
> > > Best,
> > >
> > > Nan
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <meisam.fathi@gmail.com
> >
> > > wrote:
> > >
> > > > Here are my two pennies on both designs (actor-based design vs.
> > > > single-thread polling design)
> > > >
> > > > *Single-thread polling design*
> > > > We implemented a single-thread polling mechanism for Yarn here at
> > PayPal.
> > > > Our solution is more involved because we added many new features to
> > Livy
> > > > that we had to consider when we refactored Livy's YARN interface. But
> > we
> > > > are willing to hammer our changes so it suits the need of the Livy
> > > > community best :-)
> > > >
> > > > *Actor-based design*
> > > > It seems to me that the proposed actor based design (
> > > > https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> > > > nbTQTdDFXl2XQhXDiwA/edit)
> > > > needs a few more messages and actors. Here is why.
> > > > Livy makes three (blocking) calls to YARN
> > > > 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> > > > 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which
> > gives
> > > > Livy `getAMContainerId`
> > > > 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
> > > >
> > > > The result of the previous call is needed to make the next call. The
> > > > proposed actor system needs to be designed to handles all these
> > blocking
> > > > calls.
> > > >
> > > > I do agree that actor based design is cleaner and more maintainable.
> > But
> > > we
> > > > had to discard it because it adds more dependencies to Livy. We faced
> > too
> > > > many dependency-version-mismatch problems with Livy interactive
> > sessions
> > > > (when applications depend on a different version of a library that is
> > > used
> > > > internally by Livy). If the livy community prefers an actor based
> > design,
> > > > we are willing to reimplement our changes with an actor system.
> > > >
> > > > Finally, either design is only the first step in fixing this
> particular
> > > > scalability problem. The reason is that the *"yarnAppMinotorThread"
> is
> > > not
> > > > the only thread that Livy spawns per Spark application.* For batch
> > jobs,
> > > > Livy
> > > > 1. calls spark-submit, which lunches a new JVM (an operations that is
> > far
> > > > more heavy than creating a thread and can easily drain the system)
> > > > 2. It create a thread that waits for the exist code of spark-submit.
> > Even
> > > > though this thread is "short-lived", at peak time thousands of such
> > > threads
> > > > are created in a few seconds.
> > > >
> > > > I created a PR with our modifications to it.
> > > >
> > > > https://github.com/apache/incubator-livy/pull/36
> > > >
> > > > Thanks,
> > > > Meisam
> > > >
> > > > On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <vanzin@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> > > > > <ar...@microsoft.com.invalid> wrote:
> > > > > > 1. Additional copy of states in Livy which can be queried from
> YARN
> > > on
> > > > > request.
> > > > >
> > > > > Not sure I follow.
> > > > >
> > > > > > 2. The design is not event driven and may waste querying YARN
> > > > > unnecessarily when no actual user/external request is pending.
> > > > >
> > > > > You don't need to keep querying YARN if there are no apps to
> monitor.
> > > > >
> > > > > > 3. There will always be an issue with stale data and update
> latency
> > > > > between actual YARN state and Livy state map.
> > > > >
> > > > > That is also the case with a thread pool that has less threads than
> > > > > the number of apps being monitored, if making one request per app.
> > > > >
> > > > > > 4. Size and latency of the response in bulk querying YARN is
> > unknown.
> > > > >
> > > > > That is also unknown when making multiple requests from multiple
> > > > > threads, unless you investigate the internal implementation of both
> > > > > YARN clients and servers.
> > > > >
> > > > > > 5. YARN bulk API needs to support filtering at the query level.
> > > > >
> > > > > Yes, I mentioned that in my original response and really was just
> > > > > expecting Nan to do a quick investigation of that implementation
> > > > > option.
> > > > >
> > > > > He finally did and it seems that the API only exists through the
> REST
> > > > > interface, so this all is moot.
> > > > >
> > > > > --
> > > > > Marcelo
> > > > >
> > > >
> > >
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Thanks for the answer, Meisam!

>  The time consuming parts in the code are calls to YARN and not filtering
and updating the data structures.

In the highlighted line
https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75

I assume that it will get the reports of all applications in YARN, even
they are finished?



On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <me...@gmail.com>
wrote:

> Hi Nan,
>
>
> >
> > my question related to the undergoing discussion is simply "have you seen
> > any performance issue in
> >
> > https://github.com/apache/incubator-livy/pull/36/files#diff-
> a3f879755cfe10a678cc08ddbe60a4d3R75
> > ?
> > <https://github.com/apache/incubator-livy/pull/36/files#diff-
> a3f879755cfe10a678cc08ddbe60a4d3R75?>
> > "
> >
> > The short answer is yes. This PR fixes one part of the scalability
> problem, which is, it prevents Livy from creating many
> yarnAppMinotorThreads. But the two other parts are still there
>
> 1. one call to spark-submit for each application
> 2. once thread that waits for the exit code of spark-submit.
>
> Out of these two problems, calling one spark-submit per application is the
> biggest problem, but it can be solved by adding more Livy servers. We
> modified Livy so if an application status changes on one Livy instance, all
> other Livy instances get the updated information about the application.
> From users' perspective, this is transparent because users just see the
> load balancer.
>
> So, refactoring the yarn poll mechanism + a load balancer and a grid of
> Livy servers fixed the scalability issue.
>
> On the performance of the code itself, we have not had an issue. The time
> consuming parts in the code are calls to YARN and not filtering and
> updating the data structures. On memory usage, this all needs less than 1GB
> at peak time.
>
> I hope this answers your question.
>
> Thanks,
> Meisam
>
>
> > We have several scenarios that a large volume of applications are
> submitted
> > to YARN every day and it easily accumulates a lot to be fetched with this
> > call
> >
> > Best,
> >
> > Nan
> >
> >
> >
> >
> >
> > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
> > wrote:
> >
> > > Here are my two pennies on both designs (actor-based design vs.
> > > single-thread polling design)
> > >
> > > *Single-thread polling design*
> > > We implemented a single-thread polling mechanism for Yarn here at
> PayPal.
> > > Our solution is more involved because we added many new features to
> Livy
> > > that we had to consider when we refactored Livy's YARN interface. But
> we
> > > are willing to hammer our changes so it suits the need of the Livy
> > > community best :-)
> > >
> > > *Actor-based design*
> > > It seems to me that the proposed actor based design (
> > > https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> > > nbTQTdDFXl2XQhXDiwA/edit)
> > > needs a few more messages and actors. Here is why.
> > > Livy makes three (blocking) calls to YARN
> > > 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> > > 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which
> gives
> > > Livy `getAMContainerId`
> > > 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
> > >
> > > The result of the previous call is needed to make the next call. The
> > > proposed actor system needs to be designed to handles all these
> blocking
> > > calls.
> > >
> > > I do agree that actor based design is cleaner and more maintainable.
> But
> > we
> > > had to discard it because it adds more dependencies to Livy. We faced
> too
> > > many dependency-version-mismatch problems with Livy interactive
> sessions
> > > (when applications depend on a different version of a library that is
> > used
> > > internally by Livy). If the livy community prefers an actor based
> design,
> > > we are willing to reimplement our changes with an actor system.
> > >
> > > Finally, either design is only the first step in fixing this particular
> > > scalability problem. The reason is that the *"yarnAppMinotorThread" is
> > not
> > > the only thread that Livy spawns per Spark application.* For batch
> jobs,
> > > Livy
> > > 1. calls spark-submit, which lunches a new JVM (an operations that is
> far
> > > more heavy than creating a thread and can easily drain the system)
> > > 2. It create a thread that waits for the exist code of spark-submit.
> Even
> > > though this thread is "short-lived", at peak time thousands of such
> > threads
> > > are created in a few seconds.
> > >
> > > I created a PR with our modifications to it.
> > >
> > > https://github.com/apache/incubator-livy/pull/36
> > >
> > > Thanks,
> > > Meisam
> > >
> > > On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <va...@cloudera.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> > > > <ar...@microsoft.com.invalid> wrote:
> > > > > 1. Additional copy of states in Livy which can be queried from YARN
> > on
> > > > request.
> > > >
> > > > Not sure I follow.
> > > >
> > > > > 2. The design is not event driven and may waste querying YARN
> > > > unnecessarily when no actual user/external request is pending.
> > > >
> > > > You don't need to keep querying YARN if there are no apps to monitor.
> > > >
> > > > > 3. There will always be an issue with stale data and update latency
> > > > between actual YARN state and Livy state map.
> > > >
> > > > That is also the case with a thread pool that has less threads than
> > > > the number of apps being monitored, if making one request per app.
> > > >
> > > > > 4. Size and latency of the response in bulk querying YARN is
> unknown.
> > > >
> > > > That is also unknown when making multiple requests from multiple
> > > > threads, unless you investigate the internal implementation of both
> > > > YARN clients and servers.
> > > >
> > > > > 5. YARN bulk API needs to support filtering at the query level.
> > > >
> > > > Yes, I mentioned that in my original response and really was just
> > > > expecting Nan to do a quick investigation of that implementation
> > > > option.
> > > >
> > > > He finally did and it seems that the API only exists through the REST
> > > > interface, so this all is moot.
> > > >
> > > > --
> > > > Marcelo
> > > >
> > >
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 2:58 PM, Meisam Fathi <me...@gmail.com> wrote:
> Livy can directly call org.apache.spark.deploy.SparkSubmit.main() with
> proper arguments, which is what spark-submit ends up doing.
>
> I have at least three problems with this approach:
> 1. It is a hack.
> 2. Now that you pointed out, I see it restricts Livy to a single version of
> Spark.
> 3. It becomes tricky to separate the output of different applications
> because all outputs will go to the stdout/stderr of Livy process.

That basically describes what SPARK-11035 would do. I can think of 2
ways of doing that while still supporting multiple Spark versions, one
hacky and one very hacky:

- Use different class loaders to run different Spark versions.
- Use different processes to load different Spark versions, and make
the Livy server process communicate with them when launching
applications.

Neither really solves #3. Not sure I have any good ideas of how to fix that one.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Hi Marcelo,

Thanks for your comments. I'd like to know your thoughts a different
approach.

Livy can directly call org.apache.spark.deploy.SparkSubmit.main() with
proper arguments, which is what spark-submit ends up doing.

I have at least three problems with this approach:
1. It is a hack.
2. Now that you pointed out, I see it restricts Livy to a single version of
Spark.
3. It becomes tricky to separate the output of different applications
because all outputs will go to the stdout/stderr of Livy process.

I am not sure how to fix 2. For fixing 3, I noticed that Scala has a
Console.withOut method that can be used to set "the default output stream
for the duration of execution of one thunk". I never used it though.

Thanks,
Meisam

On Wed, Aug 16, 2017 at 12:39 PM Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <me...@gmail.com>
> wrote:
> > Out of these two problems, calling one spark-submit per application is
> the
> > biggest problem, but it can be solved by adding more Livy servers.
>
> Something like SPARK-11035 could also help here. Although the
> implementation of that particular solution would probably not be a
> very good fit for Livy in its current state, since it would probably
> mean restricting the Livy server to a single version of Spark.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <me...@gmail.com> wrote:
> Out of these two problems, calling one spark-submit per application is the
> biggest problem, but it can be solved by adding more Livy servers.

Something like SPARK-11035 could also help here. Although the
implementation of that particular solution would probably not be a
very good fit for Livy in its current state, since it would probably
mean restricting the Livy server to a single version of Spark.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Hi Nan,


>
> my question related to the undergoing discussion is simply "have you seen
> any performance issue in
>
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75
> ?
> <https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75?>
> "
>
> The short answer is yes. This PR fixes one part of the scalability
problem, which is, it prevents Livy from creating many
yarnAppMinotorThreads. But the two other parts are still there

1. one call to spark-submit for each application
2. once thread that waits for the exit code of spark-submit.

Out of these two problems, calling one spark-submit per application is the
biggest problem, but it can be solved by adding more Livy servers. We
modified Livy so if an application status changes on one Livy instance, all
other Livy instances get the updated information about the application.
From users' perspective, this is transparent because users just see the
load balancer.

So, refactoring the yarn poll mechanism + a load balancer and a grid of
Livy servers fixed the scalability issue.

On the performance of the code itself, we have not had an issue. The time
consuming parts in the code are calls to YARN and not filtering and
updating the data structures. On memory usage, this all needs less than 1GB
at peak time.

I hope this answers your question.

Thanks,
Meisam


> We have several scenarios that a large volume of applications are submitted
> to YARN every day and it easily accumulates a lot to be fetched with this
> call
>
> Best,
>
> Nan
>
>
>
>
>
> On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
> wrote:
>
> > Here are my two pennies on both designs (actor-based design vs.
> > single-thread polling design)
> >
> > *Single-thread polling design*
> > We implemented a single-thread polling mechanism for Yarn here at PayPal.
> > Our solution is more involved because we added many new features to Livy
> > that we had to consider when we refactored Livy's YARN interface. But we
> > are willing to hammer our changes so it suits the need of the Livy
> > community best :-)
> >
> > *Actor-based design*
> > It seems to me that the proposed actor based design (
> > https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> > nbTQTdDFXl2XQhXDiwA/edit)
> > needs a few more messages and actors. Here is why.
> > Livy makes three (blocking) calls to YARN
> > 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> > 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which gives
> > Livy `getAMContainerId`
> > 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
> >
> > The result of the previous call is needed to make the next call. The
> > proposed actor system needs to be designed to handles all these blocking
> > calls.
> >
> > I do agree that actor based design is cleaner and more maintainable. But
> we
> > had to discard it because it adds more dependencies to Livy. We faced too
> > many dependency-version-mismatch problems with Livy interactive sessions
> > (when applications depend on a different version of a library that is
> used
> > internally by Livy). If the livy community prefers an actor based design,
> > we are willing to reimplement our changes with an actor system.
> >
> > Finally, either design is only the first step in fixing this particular
> > scalability problem. The reason is that the *"yarnAppMinotorThread" is
> not
> > the only thread that Livy spawns per Spark application.* For batch jobs,
> > Livy
> > 1. calls spark-submit, which lunches a new JVM (an operations that is far
> > more heavy than creating a thread and can easily drain the system)
> > 2. It create a thread that waits for the exist code of spark-submit. Even
> > though this thread is "short-lived", at peak time thousands of such
> threads
> > are created in a few seconds.
> >
> > I created a PR with our modifications to it.
> >
> > https://github.com/apache/incubator-livy/pull/36
> >
> > Thanks,
> > Meisam
> >
> > On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <va...@cloudera.com>
> > wrote:
> >
> > > Hello,
> > >
> > > On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> > > <ar...@microsoft.com.invalid> wrote:
> > > > 1. Additional copy of states in Livy which can be queried from YARN
> on
> > > request.
> > >
> > > Not sure I follow.
> > >
> > > > 2. The design is not event driven and may waste querying YARN
> > > unnecessarily when no actual user/external request is pending.
> > >
> > > You don't need to keep querying YARN if there are no apps to monitor.
> > >
> > > > 3. There will always be an issue with stale data and update latency
> > > between actual YARN state and Livy state map.
> > >
> > > That is also the case with a thread pool that has less threads than
> > > the number of apps being monitored, if making one request per app.
> > >
> > > > 4. Size and latency of the response in bulk querying YARN is unknown.
> > >
> > > That is also unknown when making multiple requests from multiple
> > > threads, unless you investigate the internal implementation of both
> > > YARN clients and servers.
> > >
> > > > 5. YARN bulk API needs to support filtering at the query level.
> > >
> > > Yes, I mentioned that in my original response and really was just
> > > expecting Nan to do a quick investigation of that implementation
> > > option.
> > >
> > > He finally did and it seems that the API only exists through the REST
> > > interface, so this all is moot.
> > >
> > > --
> > > Marcelo
> > >
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Hi, Meisam

Many thanks for sending the PR

my question related to the undergoing discussion is simply "have you seen
any performance issue in
https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75
?"

We have several scenarios that a large volume of applications are submitted
to YARN every day and it easily accumulates a lot to be fetched with this
call

Best,

Nan





On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
wrote:

> Here are my two pennies on both designs (actor-based design vs.
> single-thread polling design)
>
> *Single-thread polling design*
> We implemented a single-thread polling mechanism for Yarn here at PayPal.
> Our solution is more involved because we added many new features to Livy
> that we had to consider when we refactored Livy's YARN interface. But we
> are willing to hammer our changes so it suits the need of the Livy
> community best :-)
>
> *Actor-based design*
> It seems to me that the proposed actor based design (
> https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> nbTQTdDFXl2XQhXDiwA/edit)
> needs a few more messages and actors. Here is why.
> Livy makes three (blocking) calls to YARN
> 1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
> 2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which gives
> Livy `getAMContainerId`
> 3. `yarnClient.getContainerReport`, which gives Livy tracking URLs
>
> The result of the previous call is needed to make the next call. The
> proposed actor system needs to be designed to handles all these blocking
> calls.
>
> I do agree that actor based design is cleaner and more maintainable. But we
> had to discard it because it adds more dependencies to Livy. We faced too
> many dependency-version-mismatch problems with Livy interactive sessions
> (when applications depend on a different version of a library that is used
> internally by Livy). If the livy community prefers an actor based design,
> we are willing to reimplement our changes with an actor system.
>
> Finally, either design is only the first step in fixing this particular
> scalability problem. The reason is that the *"yarnAppMinotorThread" is not
> the only thread that Livy spawns per Spark application.* For batch jobs,
> Livy
> 1. calls spark-submit, which lunches a new JVM (an operations that is far
> more heavy than creating a thread and can easily drain the system)
> 2. It create a thread that waits for the exist code of spark-submit. Even
> though this thread is "short-lived", at peak time thousands of such threads
> are created in a few seconds.
>
> I created a PR with our modifications to it.
>
> https://github.com/apache/incubator-livy/pull/36
>
> Thanks,
> Meisam
>
> On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
> > Hello,
> >
> > On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> > <ar...@microsoft.com.invalid> wrote:
> > > 1. Additional copy of states in Livy which can be queried from YARN on
> > request.
> >
> > Not sure I follow.
> >
> > > 2. The design is not event driven and may waste querying YARN
> > unnecessarily when no actual user/external request is pending.
> >
> > You don't need to keep querying YARN if there are no apps to monitor.
> >
> > > 3. There will always be an issue with stale data and update latency
> > between actual YARN state and Livy state map.
> >
> > That is also the case with a thread pool that has less threads than
> > the number of apps being monitored, if making one request per app.
> >
> > > 4. Size and latency of the response in bulk querying YARN is unknown.
> >
> > That is also unknown when making multiple requests from multiple
> > threads, unless you investigate the internal implementation of both
> > YARN clients and servers.
> >
> > > 5. YARN bulk API needs to support filtering at the query level.
> >
> > Yes, I mentioned that in my original response and really was just
> > expecting Nan to do a quick investigation of that implementation
> > option.
> >
> > He finally did and it seems that the API only exists through the REST
> > interface, so this all is moot.
> >
> > --
> > Marcelo
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 11:27 AM, Nan Zhu <zh...@gmail.com> wrote:
> yes, it is going to be Akka if moving forward (at least not going to
> introduce an actor framework to livy)

-1 on that. I don't see a reason to introduce a large and complex
framework like Akka into Livy.

What you propose can be achieved easily with existing Java libraries.


> On Wed, Aug 16, 2017 at 11:24 AM, Meisam Fathi <me...@gmail.com>
> wrote:
>
>> That is true, but I was under the impression that this will be implemented
>> with Akka (maybe because it is mentioned in the design doc).
>>
>> On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>>
>> > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
>> > wrote:
>> > > I do agree that actor based design is cleaner and more maintainable.
>> But
>> > we
>> > > had to discard it because it adds more dependencies to Livy.
>> >
>> > I've been reading "actor system" as a design pattern, not as
>> > introducing a new dependency to Livy.
>> >
>> > If the document is actually proposing using Akka (instead of just
>> > using Akka as an example of an actor system implementation), then I'm
>> > a -1 on that.
>> >
>> > --
>> > Marcelo
>> >
>>



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

yes, it is going to be Akka if moving forward (at least not going to
introduce an actor framework to livy)

On Wed, Aug 16, 2017 at 11:24 AM, Meisam Fathi <me...@gmail.com>
wrote:

> That is true, but I was under the impression that this will be implemented
> with Akka (maybe because it is mentioned in the design doc).
>
> On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
> > On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
> > wrote:
> > > I do agree that actor based design is cleaner and more maintainable.
> But
> > we
> > > had to discard it because it adds more dependencies to Livy.
> >
> > I've been reading "actor system" as a design pattern, not as
> > introducing a new dependency to Livy.
> >
> > If the document is actually proposing using Akka (instead of just
> > using Akka as an example of an actor system implementation), then I'm
> > a -1 on that.
> >
> > --
> > Marcelo
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

That is true, but I was under the impression that this will be implemented
with Akka (maybe because it is mentioned in the design doc).

On Wed, Aug 16, 2017 at 11:21 AM Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com>
> wrote:
> > I do agree that actor based design is cleaner and more maintainable. But
> we
> > had to discard it because it adds more dependencies to Livy.
>
> I've been reading "actor system" as a design pattern, not as
> introducing a new dependency to Livy.
>
> If the document is actually proposing using Akka (instead of just
> using Akka as an example of an actor system implementation), then I'm
> a -1 on that.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 11:16 AM, Meisam Fathi <me...@gmail.com> wrote:
> I do agree that actor based design is cleaner and more maintainable. But we
> had to discard it because it adds more dependencies to Livy.

I've been reading "actor system" as a design pattern, not as
introducing a new dependency to Livy.

If the document is actually proposing using Akka (instead of just
using Akka as an example of an actor system implementation), then I'm
a -1 on that.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Here are my two pennies on both designs (actor-based design vs.
single-thread polling design)

*Single-thread polling design*
We implemented a single-thread polling mechanism for Yarn here at PayPal.
Our solution is more involved because we added many new features to Livy
that we had to consider when we refactored Livy's YARN interface. But we
are willing to hammer our changes so it suits the need of the Livy
community best :-)

*Actor-based design*
It seems to me that the proposed actor based design (
https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit)
needs a few more messages and actors. Here is why.
Livy makes three (blocking) calls to YARN
1. `yarnClient.getApplications`, which gives Livy `ApplicatioId`s
2. `yarnClient.getApplicationAttemptReport(ApplicationId)`, which gives
Livy `getAMContainerId`
3. `yarnClient.getContainerReport`, which gives Livy tracking URLs

The result of the previous call is needed to make the next call. The
proposed actor system needs to be designed to handles all these blocking
calls.

I do agree that actor based design is cleaner and more maintainable. But we
had to discard it because it adds more dependencies to Livy. We faced too
many dependency-version-mismatch problems with Livy interactive sessions
(when applications depend on a different version of a library that is used
internally by Livy). If the livy community prefers an actor based design,
we are willing to reimplement our changes with an actor system.

Finally, either design is only the first step in fixing this particular
scalability problem. The reason is that the *"yarnAppMinotorThread" is not
the only thread that Livy spawns per Spark application.* For batch jobs,
Livy
1. calls spark-submit, which lunches a new JVM (an operations that is far
more heavy than creating a thread and can easily drain the system)
2. It create a thread that waits for the exist code of spark-submit. Even
though this thread is "short-lived", at peak time thousands of such threads
are created in a few seconds.

I created a PR with our modifications to it.

https://github.com/apache/incubator-livy/pull/36

Thanks,
Meisam

On Wed, Aug 16, 2017 at 10:42 AM Marcelo Vanzin <va...@cloudera.com> wrote:

> Hello,
>
> On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
> <ar...@microsoft.com.invalid> wrote:
> > 1. Additional copy of states in Livy which can be queried from YARN on
> request.
>
> Not sure I follow.
>
> > 2. The design is not event driven and may waste querying YARN
> unnecessarily when no actual user/external request is pending.
>
> You don't need to keep querying YARN if there are no apps to monitor.
>
> > 3. There will always be an issue with stale data and update latency
> between actual YARN state and Livy state map.
>
> That is also the case with a thread pool that has less threads than
> the number of apps being monitored, if making one request per app.
>
> > 4. Size and latency of the response in bulk querying YARN is unknown.
>
> That is also unknown when making multiple requests from multiple
> threads, unless you investigate the internal implementation of both
> YARN clients and servers.
>
> > 5. YARN bulk API needs to support filtering at the query level.
>
> Yes, I mentioned that in my original response and really was just
> expecting Nan to do a quick investigation of that implementation
> option.
>
> He finally did and it seems that the API only exists through the REST
> interface, so this all is moot.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

Hello,

On Wed, Aug 16, 2017 at 10:35 AM, Arijit Tarafdar
<ar...@microsoft.com.invalid> wrote:
> 1. Additional copy of states in Livy which can be queried from YARN on request.

Not sure I follow.

> 2. The design is not event driven and may waste querying YARN unnecessarily when no actual user/external request is pending.

You don't need to keep querying YARN if there are no apps to monitor.

> 3. There will always be an issue with stale data and update latency between actual YARN state and Livy state map.

That is also the case with a thread pool that has less threads than
the number of apps being monitored, if making one request per app.

> 4. Size and latency of the response in bulk querying YARN is unknown.

That is also unknown when making multiple requests from multiple
threads, unless you investigate the internal implementation of both
YARN clients and servers.

> 5. YARN bulk API needs to support filtering at the query level.

Yes, I mentioned that in my original response and really was just
expecting Nan to do a quick investigation of that implementation
option.

He finally did and it seems that the API only exists through the REST
interface, so this all is moot.

-- 
Marcelo

RE: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Arijit Tarafdar <ar...@microsoft.com.INVALID>]

We need the following scenarios to be supported:

1. Individual application submission.
2. Individual application status query.
3. Batch application status query.
4. Batch application status query by application status.
5. Batch application status query by application user.
6. Batch application status query by application user and status.

More application status query filters can be added but let's stop at that. Also we need to add throttling on top of it whose absence in LIVY creates lots of issues in production (that is for a subsequent discussion).

@Marcelo
Now given those requirements suppose we have an YARN API to bulk query the application and which also supports filtering are you proposing that Livy maintains the application state map (as cache) which will be updated by a single thread at regular interval by querying YARN? If true, the problems I see are the following:

1. Additional copy of states in Livy which can be queried from YARN on request.
2. The design is not event driven and may waste querying YARN unnecessarily when no actual user/external request is pending.
3. There will always be an issue with stale data and update latency between actual YARN state and Livy state map.
4. Size and latency of the response in bulk querying YARN is unknown.
5. YARN bulk API needs to support filtering at the query level.

Thanks, Arijit

-----Original Message-----
From: Nan Zhu [mailto:zhunanmcgill@gmail.com] 
Sent: Wednesday, August 16, 2017 10:31 AM
To: dev@livy.incubator.apache.org
Subject: Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

> I really don't understand what you mean. You need somewhere to keep 
> the
application handles you're monitoring regarding of the solution. The code making the YARN request needs to somehow update those handles. Whether there's a task per handle that is submitted to a thread pool, or some map or list tracking all available handles that are then updated by the single thread talking to YARN, it doesn't matter.

> In the first case your thread pool is the "shared data structure", in 
> the
second case this map of handles is the "shared data structure", so I don't understand why you think there is any difference.

I do not understand why there is no difference

In your words, when the thread pool is the "shared data structure", we do not need to involve any synchronization when applying CRUD to the handles.
If you shared "some map or list" between servlet threads and monitoring thread, you have to handle synchronization of this "map or list". And yes, if you have single monitoring thread, life can be easier (the potential cons about single thread to handle everything through bulk operation is another topic in this email)

> I'm proposing a different approach that I'm pretty sure is easier on 
> YARN,
which is a shared service that we should be trying not to unnecessarily overload. The least I'd expect is for you to consider the suggestion and actually explain why it wouldn't work, but so far you've just been deflecting feedback.

> You can, for example, see if such a bulk API exists and reply "I 
> couldn't
find it". I believe it must exist, after all I can go to the RM web UI and see all applications, and get a list of them from the YARN REST API. But if it doesn't exist, that would take care of my suggestion.

You mixed two topics again

Topic 1 - what you proposed: I keep trying to discuss about the pros & cons of single thread model, and I have said it for multiple times, it can make life easier, but with additional efforts on 1. synchronization over a map/list, 2. handling of exceptions (due to the dependency among all running apps), etc.

Topic 2 - How we discuss: I am surprised that you blamed my way to discuss even before you correct your own attitude. When we are conducting technical discussion, I believe you are not in a good position to make it health, e.g. when I said multiple thread can share the same RPC connections to avoid your concern about multiple tasks will keep opening/closing connections, you replied "Irrelevant" without any explanation, and also, I didn't see any concrete evidence from you about why Actor-based solution is not an option. On the other side, I have show you my concerns on bulk operations, which are worth more discussions even though until so far I just received some feedback like "Irrelevant" or "how about datacenter is down"


> "I would investigate" is a suggestion that you investigate that as 
> part
of proposing your change. It's not me saying that I'll do it myself (that would be "I will investigate").

OK, I found it,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fstable%2Fhadoop-yarn%2Fhadoop-yarn-site%2FResourceManagerRest.html%23Cluster_Applications_API&data=02%7C01%7Carijitt%40microsoft.com%7C26e05599670540cc08b808d4e4cca4c7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636385014975825054&sdata=4jnMFOqIF4Pku30O30QCLIx29OTEGafB%2FTj3hvyg9pI%3D&reserved=0,
which is a restful API,

two concerns

1. change Livy's current impl is based on YarnClient not restful API, are we going to change it?

2. being different with RM UI which can only fetch 20 applications for every page, we need to fetch all applications (since it looks like we cannot do "not match" against application state through this API)


> I'm expecting that errors be handled regardless of the situation. If 
> YARN
returns an error to you, regardless of whether it was a request for a single application status or for a bunch of them, your code needs to handle it somehow. The handling will most probably be the same in both cases (retry), and that's my point.

yes, this is one of the possible solutions, the pros is it is simple and easy to handle, the cons is that it makes whether application A's state is stale depend on all the other applications, which also needs more discussions


Again, please make technical discussion as professional as possible



On Wed, Aug 16, 2017 at 9:44 AM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 9:33 AM, Nan Zhu <zh...@gmail.com> wrote:
> >> What I proposed is having a single request to YARN to get all
> applications'
> > statuses, if that's possible. You'd still have multiple application
> handles
> > that are independent of each other. They'd all be updated separately 
> > from that one thread talking to YARN. This has nothing to do with a 
> > "shared
> data
> > structure". There's no shared data structure here to track 
> > application status.
> >
> > You are still avoiding the questions how you make all "application
> handles"
> > accessible to this thread
>
> I really don't understand what you mean. You need somewhere to keep 
> the application handles you're monitoring regarding of the solution.
> The code making the YARN request needs to somehow update those 
> handles. Whether there's a task per handle that is submitted to a 
> thread pool, or some map or list tracking all available handles that 
> are then updated by the single thread talking to YARN, it doesn't 
> matter.
>
> In the first case your thread pool is the "shared data structure", in 
> the second case this map of handles is the "shared data structure", so 
> I don't understand why you think there is any difference.
>
> I'm proposing a different approach that I'm pretty sure is easier on 
> YARN, which is a shared service that we should be trying not to 
> unnecessarily overload. The least I'd expect is for you to consider 
> the suggestion and actually explain why it wouldn't work, but so far 
> you've just been deflecting feedback.
>
> You can, for example, see if such a bulk API exists and reply "I 
> couldn't find it". I believe it must exist, after all I can go to the 
> RM web UI and see all applications, and get a list of them from the 
> YARN REST API. But if it doesn't exist, that would take care of my 
> suggestion.
>
> > "I would investigate whether there's any API in YARN to do a bulk 
> > get of running applications with a particular filter;" - from your 
> > email
> >
> > If you suggest something, please find evidence to support you
>
> "I would investigate" is a suggestion that you investigate that as 
> part of proposing your change. It's not me saying that I'll do it 
> myself (that would be "I will investigate").
>
> >> What if YARN goes down? What if your datacenter has a massive power
> > failure? You have to handle errors in any scenario.
> >
> > Again, I am describing one concrete scenario which is always 
> > involved in any bulk operation and even we go to bulk direction, you 
> > have to handle this. Since you proposed this bulk operation, I am 
> > asking you what's your expectation about this.
>
> I'm expecting that errors be handled regardless of the situation. If 
> YARN returns an error to you, regardless of whether it was a request 
> for a single application status or for a bunch of them, your code 
> needs to handle it somehow. The handling will most probably be the 
> same in both cases (retry), and that's my point.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 2:09 PM, Nan Zhu <zh...@gmail.com> wrote:
> With time goes, the reply from YARN can only be larger and larger. Given
> the consistent workload pattern, the cost of a large query can be
> eventually larger than individual request

That's where filtering would help, if it's possible to do it easily.

> I would say go with individual request + thread pool  or large batch for
> all first, if any performance issue is observed, add the optimization on
> top of it

How about doing some experiments?

You seem to have spent time with your proposed approach, so I believe
there's at least some kind of prototype you're working on. It should
be easy to get average latency for each request and throughput for
different thread counts.

You can do a crude approximation of what it would take to get the same
data in bulk by hitting the REST API with curl; no need to write code,
and you get both an idea of latency and of the size of the bulk
replies.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Saisai Shao <sa...@gmail.com>]

Nan, I think Meisam already had a PR about this this, maybe you can discuss
with him on the github based on the proposed code.

Sorry I didn't follow the long discussion thread, but I think Paypal's
solution sounds simpler.

On Wed, Aug 23, 2017 at 12:07 AM, Nan Zhu <zh...@gmail.com> wrote:

> based on this result, I think we should follow the bulk operation pattern
>
> Shall we move forward with the PR from Paypal?
>
> Best,
>
> Nan
>
> On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi <me...@gmail.com>
> wrote:
>
> > Bottom line up front:
> > 1. The cost of calling 10000 individual REST calls is about two order of
> > magnitude higher than calling a single batch REST call (10000 * 0.05
> > seconds vs. 1.4 seconds)
> > 2. Time to complete a batch REST call plateaus at about 10,000
> application
> > reports per call.
> >
> > Full story:
> > I experimented and measure how long it takes to fetch Application Reports
> > from YARN with the REST API. My objective was to compare doing a batch
> REST
> > call to get all ApplicationReports vs doing individual REST calls for
> each
> > Application Report.
> >
> > I did the tests on 4 different cluster: 1) a test cluster, 2) a
> moderately
> > used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> > used production cluster. For each cluster I made 7 REST call to get 1,
> 10,
> > 100, 1000, 10000, 100000, 1000000 application reports respectively. I
> > repeated each call 200 times to count for variations and I reported the
> > median time.
> > To measure the time, I used the following curl command:
> >
> > $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> > $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> > applicationTypes&limit=$limit"
> >
> > The attached charts show the results. In all the charts, the x axis show
> > the number of results that were request in the call.
> > The bar chart show the time it takes to complete a REST call on each
> > cluster.
> > The first line plot also shows the same results as the bar chart on a log
> > scale (it is easier to see that the time to complete the REST call
> plateaus
> > at 10,000
> > The last chart shows the size of data that is being downloaded on each
> > REST call, which explains why the time plateaus  at 10,000.
> >
> >
> > [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][
> image:
> > size_downloaded_line_plot.png]
> >
> >>
> >>
> > Thanks,
> > Meisam
> >
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

based on this result, I think we should follow the bulk operation pattern

Shall we move forward with the PR from Paypal?

Best,

Nan

On Mon, Aug 21, 2017 at 12:21 PM, Meisam Fathi <me...@gmail.com>
wrote:

> Bottom line up front:
> 1. The cost of calling 10000 individual REST calls is about two order of
> magnitude higher than calling a single batch REST call (10000 * 0.05
> seconds vs. 1.4 seconds)
> 2. Time to complete a batch REST call plateaus at about 10,000 application
> reports per call.
>
> Full story:
> I experimented and measure how long it takes to fetch Application Reports
> from YARN with the REST API. My objective was to compare doing a batch REST
> call to get all ApplicationReports vs doing individual REST calls for each
> Application Report.
>
> I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
> used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> used production cluster. For each cluster I made 7 REST call to get 1, 10,
> 100, 1000, 10000, 100000, 1000000 application reports respectively. I
> repeated each call 200 times to count for variations and I reported the
> median time.
> To measure the time, I used the following curl command:
>
> $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$
> applicationTypes&limit=$limit"
>
> The attached charts show the results. In all the charts, the x axis show
> the number of results that were request in the call.
> The bar chart show the time it takes to complete a REST call on each
> cluster.
> The first line plot also shows the same results as the bar chart on a log
> scale (it is easier to see that the time to complete the REST call plateaus
> at 10,000
> The last chart shows the size of data that is being downloaded on each
> REST call, which explains why the time plateaus  at 10,000.
>
>
> [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
> size_downloaded_line_plot.png]
>
>>
>>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Hi Marcelo,


> I'm not really familiar with how multi-node HA was implemented (I
> stopped at session recovery), but why isn't a single server doing the
> update and storing the results in ZK? Unless it's actually doing
> load-balancing, it seems like that would avoid multiple servers having
> to hit YARN.
>

We considered having one server update ZooKeeper, but the extra benefits
that we would get from polling yarn fewer times is not worth the extra
complexity needed to implement it. For example, we would have to make
servers aware of each other, and aware of each others failures. We would've
needed a voting mechanism to select a new leader to update ZooKeeper each
time the current leader had a failure. Also rolling out updates would be
tricker with servers that are aware of each other.

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

> Just an FYI, apache mailing lists cant share attachments. If you could
> please upload the files to another file sharing site and include links
> instead.
>
Thanks for the information. I added the files to the JIRA ticket and put
the contents of the previous email as a comment. Here are the links to the
ticket and to the files:

JIRA ticket: https://issues.apache.org/jira/browse/LIVY-336
Time to complete REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882985/transfer_time_bar_plot.png
Trends in time to complete REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882984/transfer_time_line_plot.png
Size of response from REST calls to YARN:
https://issues.apache.org/jira/secure/attachment/12882983/size_downloaded_line_plot.png

Also, should we move the discussion to JIRA now that it is up and running?

Thanks,
Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Alex Bozarth <aj...@us.ibm.com>]

Just an FYI, apache mailing lists cant share attachments. If you could
please upload the files to another file sharing site and include links
instead.

Thanks,
                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 E-mail: ajbozart@us.ibm.com                                                         
 GitHub: github.com/ajbozarth                                                        
                                                                   505 Howard Street 
                                                             San Francisco, CA 94105 
                                                                       United States 
                                                                                     








From:	Meisam Fathi <me...@gmail.com>
To:	dev@livy.incubator.apache.org
Cc:	Prabhu Kasinathan <va...@gmail.com>
Date:	08/21/2017 02:09 PM
Subject:	Re: resolve the scalability problem caused by app monitoring in
            livy with an actor-based design



I forgot to attach the first chart. Sorry about that.



Thanks,
Meisam

On Mon, Aug 21, 2017 at 12:21 PM Meisam Fathi <me...@gmail.com>
wrote:
  Bottom line up front:
  1. The cost of calling 10000 individual REST calls is about two order of
  magnitude higher than calling a single batch REST call (10000 * 0.05
  seconds vs. 1.4 seconds)
  2. Time to complete a batch REST call plateaus at about 10,000
  application reports per call.

  Full story:
  I experimented and measure how long it takes to fetch Application Reports
  from YARN with the REST API. My objective was to compare doing a batch
  REST call to get all ApplicationReports vs doing individual REST calls
  for each Application Report.

  I did the tests on 4 different cluster: 1) a test cluster, 2) a
  moderately used dev cluster, 3) a lightly used production cluster, and 4)
  a heavily used production cluster. For each cluster I made 7 REST call to
  get 1, 10, 100, 1000, 10000, 100000, 1000000 application reports
  respectively. I repeated each call 200 times to count for variations and
  I reported the median time.
  To measure the time, I used the following curl command:

  $ curl -o /dev/null -s -w "@curl-output-fromat.json" "
  http://$rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=
  $applicationTypes&limit=$limit"

  The attached charts show the results. In all the charts, the x axis show
  the number of results that were request in the call.
  The bar chart show the time it takes to complete a REST call on each
  cluster.
  The first line plot also shows the same results as the bar chart on a log
  scale (it is easier to see that the time to complete the REST call
  plateaus at 10,000
  The last chart shows the size of data that is being downloaded on each
  REST call, which explains why the time plateaus  at 10,000.





  Thanks,
  Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

I forgot to attach the first chart. Sorry about that.

[image: transfer_time_bar_plot.png]

Thanks,
Meisam

On Mon, Aug 21, 2017 at 12:21 PM Meisam Fathi <me...@gmail.com>
wrote:

> Bottom line up front:
> 1. The cost of calling 10000 individual REST calls is about two order of
> magnitude higher than calling a single batch REST call (10000 * 0.05
> seconds vs. 1.4 seconds)
> 2. Time to complete a batch REST call plateaus at about 10,000 application
> reports per call.
>
> Full story:
> I experimented and measure how long it takes to fetch Application Reports
> from YARN with the REST API. My objective was to compare doing a batch REST
> call to get all ApplicationReports vs doing individual REST calls for each
> Application Report.
>
> I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
> used dev cluster, 3) a lightly used production cluster, and 4) a heavily
> used production cluster. For each cluster I made 7 REST call to get 1, 10,
> 100, 1000, 10000, 100000, 1000000 application reports respectively. I
> repeated each call 200 times to count for variations and I reported the
> median time.
> To measure the time, I used the following curl command:
>
> $ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
> $rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$applicationTypes&limit=$limit"
>
> The attached charts show the results. In all the charts, the x axis show
> the number of results that were request in the call.
> The bar chart show the time it takes to complete a REST call on each
> cluster.
> The first line plot also shows the same results as the bar chart on a log
> scale (it is easier to see that the time to complete the REST call plateaus
> at 10,000
> The last chart shows the size of data that is being downloaded on each
> REST call, which explains why the time plateaus  at 10,000.
>
>
> [image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
> size_downloaded_line_plot.png]
>
>>
>>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

Bottom line up front:
1. The cost of calling 10000 individual REST calls is about two order of
magnitude higher than calling a single batch REST call (10000 * 0.05
seconds vs. 1.4 seconds)
2. Time to complete a batch REST call plateaus at about 10,000 application
reports per call.

Full story:
I experimented and measure how long it takes to fetch Application Reports
from YARN with the REST API. My objective was to compare doing a batch REST
call to get all ApplicationReports vs doing individual REST calls for each
Application Report.

I did the tests on 4 different cluster: 1) a test cluster, 2) a moderately
used dev cluster, 3) a lightly used production cluster, and 4) a heavily
used production cluster. For each cluster I made 7 REST call to get 1, 10,
100, 1000, 10000, 100000, 1000000 application reports respectively. I
repeated each call 200 times to count for variations and I reported the
median time.
To measure the time, I used the following curl command:

$ curl -o /dev/null -s -w "@curl-output-fromat.json" "http://
$rm_http_address:$rm_port/ws/v1/cluster/apps?applicationTypes=$applicationTypes&limit=$limit"

The attached charts show the results. In all the charts, the x axis show
the number of results that were request in the call.
The bar chart show the time it takes to complete a REST call on each
cluster.
The first line plot also shows the same results as the bar chart on a log
scale (it is easier to see that the time to complete the REST call plateaus
at 10,000
The last chart shows the size of data that is being downloaded on each REST
call, which explains why the time plateaus  at 10,000.


[image: transfer_time_bar_plot.png][image: transfer_time_line_plot.png][image:
size_downloaded_line_plot.png]

>
>
Thanks,
Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

I like that approach on paper, although I currently don't have much
time to actually be able to review the PR and provide decent feedback.

I think that regardless of the approach, one goal should be to
probably separate what is being monitored from how it's being
monitored; that way you can later change the monitoring code to be
smarter without having to change the rest of the code that calls into
it. I remember reviewing this code when it first was submitted and it
could definitely use some refactoring.

> we have Livy Multi-Node HA i.e livy running on 6 servers for each cluster,

I'm not really familiar with how multi-node HA was implemented (I
stopped at session recovery), but why isn't a single server doing the
update and storing the results in ZK? Unless it's actually doing
load-balancing, it seems like that would avoid multiple servers having
to hit YARN.



On Wed, Aug 16, 2017 at 4:18 PM, Prabhu Kasinathan
<va...@gmail.com> wrote:
> As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
> running on 6 servers for each cluster, load-balanced, sharing livy metadata
> on zookeeper and running thousands of applications. With below changes, we
> are seeing good improvements due to batching the requests (one per livy
> node) instead of each livy node making multiple requests. Please review the
> changes and let us know if improvements needed or we are open to explore
> other alternative option if works.
>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>
>
>
> On Wed, Aug 16, 2017 at 3:33 PM, Meisam Fathi <me...@gmail.com>
> wrote:
>
>>
>> On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu <zh...@gmail.com> wrote:
>>
>>> With time goes, the reply from YARN can only be larger and larger. Given
>>> the consistent workload pattern, the cost of a large query can be
>>> eventually larger than individual request
>>>
>>
>> I am under the impression that there is a limit to the number of reports
>> that YARN retains, which is set by yarn.resourcemanager.max-completed-applications
>> in yarn.xml and defaults to 10,000. But I could be wrong about the
>> semantics of yarn.resourcemanager.max-completed-applications.
>>
>> I would say go with individual request + thread pool  or large batch for
>>> all first, if any performance issue is observed, add the optimization on
>>> top of it
>>>
>>
>> We are making one big request to get ApplicationReports, Then we make an
>> individual + thread pool request to get the tracking URL, Spark UI URL,
>> YARN diagnostics, etc for each application separately. For our cluster
>> settings and our workloads, one big request turned out to be a better
>> solution. But we were limited to the API provided in YarnClient. With the
>> home-made REST client a separate request is not needed and that can change
>> the whole equation.
>>
>> @Prabhu, can you chime in?
>>
>>
>>> However, even with rest API, there are some corner cases, e.g. a
>>> long running app lasting for days (training some models), and some short
>>> ones which last only for minutes
>>>
>>
>> We are running Spark streaming jobs on Livy that virtually run for ever.
>>
>> Thanks,
>> Meisam
>>



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Prabhu Kasinathan <va...@gmail.com>]

As Meisam highlighted, in our case, we have Livy Multi-Node HA i.e livy
running on 6 servers for each cluster, load-balanced, sharing livy metadata
on zookeeper and running thousands of applications. With below changes, we
are seeing good improvements due to batching the requests (one per livy
node) instead of each livy node making multiple requests. Please review the
changes and let us know if improvements needed or we are open to explore
other alternative option if works.

> We are making one big request to get ApplicationReports, Then we make an
> individual + thread pool request to get the tracking URL, Spark UI URL,
> YARN diagnostics, etc for each application separately. For our cluster
> settings and our workloads, one big request turned out to be a better
> solution. But we were limited to the API provided in YarnClient. With the
> home-made REST client a separate request is not needed and that can change
> the whole equation.



On Wed, Aug 16, 2017 at 3:33 PM, Meisam Fathi <me...@gmail.com>
wrote:

>
> On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu <zh...@gmail.com> wrote:
>
>> With time goes, the reply from YARN can only be larger and larger. Given
>> the consistent workload pattern, the cost of a large query can be
>> eventually larger than individual request
>>
>
> I am under the impression that there is a limit to the number of reports
> that YARN retains, which is set by yarn.resourcemanager.max-completed-applications
> in yarn.xml and defaults to 10,000. But I could be wrong about the
> semantics of yarn.resourcemanager.max-completed-applications.
>
> I would say go with individual request + thread pool  or large batch for
>> all first, if any performance issue is observed, add the optimization on
>> top of it
>>
>
> We are making one big request to get ApplicationReports, Then we make an
> individual + thread pool request to get the tracking URL, Spark UI URL,
> YARN diagnostics, etc for each application separately. For our cluster
> settings and our workloads, one big request turned out to be a better
> solution. But we were limited to the API provided in YarnClient. With the
> home-made REST client a separate request is not needed and that can change
> the whole equation.
>
> @Prabhu, can you chime in?
>
>
>> However, even with rest API, there are some corner cases, e.g. a
>> long running app lasting for days (training some models), and some short
>> ones which last only for minutes
>>
>
> We are running Spark streaming jobs on Livy that virtually run for ever.
>
> Thanks,
> Meisam
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Meisam Fathi <me...@gmail.com>]

On Wed, Aug 16, 2017 at 2:09 PM Nan Zhu <zh...@gmail.com> wrote:

> With time goes, the reply from YARN can only be larger and larger. Given
> the consistent workload pattern, the cost of a large query can be
> eventually larger than individual request
>

I am under the impression that there is a limit to the number of reports
that YARN retains, which is set
by yarn.resourcemanager.max-completed-applications in yarn.xml and defaults
to 10,000. But I could be wrong about the semantics of
yarn.resourcemanager.max-completed-applications.

I would say go with individual request + thread pool  or large batch for
> all first, if any performance issue is observed, add the optimization on
> top of it
>

We are making one big request to get ApplicationReports, Then we make an
individual + thread pool request to get the tracking URL, Spark UI URL,
YARN diagnostics, etc for each application separately. For our cluster
settings and our workloads, one big request turned out to be a better
solution. But we were limited to the API provided in YarnClient. With the
home-made REST client a separate request is not needed and that can change
the whole equation.

@Prabhu, can you chime in?


> However, even with rest API, there are some corner cases, e.g. a
> long running app lasting for days (training some models), and some short
> ones which last only for minutes
>

We are running Spark streaming jobs on Livy that virtually run for ever.

Thanks,
Meisam

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

With time goes, the reply from YARN can only be larger and larger. Given
the consistent workload pattern, the cost of a large query can be
eventually larger than individual request

I would say go with individual request + thread pool  or large batch for
all first, if any performance issue is observed, add the optimization on
top of it

Regarding how to optimize,

The major issue is that  YarnClient API is a simplified version of Rest
APIs regarding the less number of filtering parameters.

I looked at the usage of YarnClient in the current implementation (only
Livy-Server), only SparkYarnApp class is using that. Since there will be a
big refactoring of this class, replacing YarnClient with a home-made
Restful Client might not be that costly


*multiple Individual request:*

Batching individual requests based on submission time

*a single Large request:*

Limiting number of fetched app status can be achieved with, e.g.
application submission time, or limit.....which are only available with
rest APIs. However, even with rest API, there are some corner cases, e.g. a
long running app lasting for days (training some models), and some short
ones which last only for minutes

Best,

Nan



On Wed, Aug 16, 2017 at 1:01 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 12:57 PM, Nan Zhu <zh...@gmail.com> wrote:
> > yes, we finally converge on the idea
> >
> > how large the reply can be? if I have only one running applications and I
> > still need to fetch 1000
> >
> > on the other side
> >
> > I have 1000 running apps, what's the cost of sending 1000 requests even
> the
> > thread pool and yarn client are shared?
>
> I don't know the answers, but I'm asking you, since you are proposing
> the design, to consider that as an option, since it does not seem like
> you considered that tradeoff when suggesting your current approach.
>
> My comments about filtering are targeted at making things better in
> your first case; if there's really only one app being monitored, and
> you can figure out a filter that returns let's say 50 apps instead of
> 1000 that may be monitored by YARN, then you can do that.
>
> Or maybe you can go with a hybrid approach, where you use individual
> requests but past a certain threshold you fall back to bulk requests
> to avoid overloading YARN.
>
> Again, I'm asking you to consider alternatives that are not mentioned
> in your design document, because I identified potential performance
> issues in the current approach.
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 12:57 PM, Nan Zhu <zh...@gmail.com> wrote:
> yes, we finally converge on the idea
>
> how large the reply can be? if I have only one running applications and I
> still need to fetch 1000
>
> on the other side
>
> I have 1000 running apps, what's the cost of sending 1000 requests even the
> thread pool and yarn client are shared?

I don't know the answers, but I'm asking you, since you are proposing
the design, to consider that as an option, since it does not seem like
you considered that tradeoff when suggesting your current approach.

My comments about filtering are targeted at making things better in
your first case; if there's really only one app being monitored, and
you can figure out a filter that returns let's say 50 apps instead of
1000 that may be monitored by YARN, then you can do that.

Or maybe you can go with a hybrid approach, where you use individual
requests but past a certain threshold you fall back to bulk requests
to avoid overloading YARN.

Again, I'm asking you to consider alternatives that are not mentioned
in your design document, because I identified potential performance
issues in the current approach.


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

yes, we finally converge on the idea

how large the reply can be? if I have only one running applications and I
still need to fetch 1000

on the other side

I have 1000 running apps, what's the cost of sending 1000 requests even the
thread pool and yarn client are shared?



On Wed, Aug 16, 2017 at 12:36 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu <zh...@gmail.com> wrote:
> > I am using your words *current*. What's the definition of "current" in
> > livy? I think that's all application which still keep some records in the
> > livy's process's memory space
>
> There are two views of what is current: Livy's and YARN's. They may
> not be the same.
>
> From your reply below, you seem to want to query YARN for the state of
> applications that are current to Livy. There's no API for that, as you
> said. But that is not what I'm talking about.
>
> I'm saying that Livy should query YARN for YARN's current view of what
> applications exist, and then match those against its own view.
>
> Again, it's all a question about what is cheaper: a single request to
> YARN that results in a large reply, parts of which Livy will ignore
> because it's not interested in the data, or hundreds of small requests
> to YARN polling specific applications?
>
> > 1. How you express this "current" in a query to YARN? I think you have to
> > use ApplicationID (maybe there are some other ways) in a query
> >
> > 2. The problem is that I didn't see such an API to make such a "big call"
> > by passing in all applications's IDs
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 12:27 PM, Nan Zhu <zh...@gmail.com> wrote:
> I am using your words *current*. What's the definition of "current" in
> livy? I think that's all application which still keep some records in the
> livy's process's memory space

There are two views of what is current: Livy's and YARN's. They may
not be the same.

From your reply below, you seem to want to query YARN for the state of
applications that are current to Livy. There's no API for that, as you
said. But that is not what I'm talking about.

I'm saying that Livy should query YARN for YARN's current view of what
applications exist, and then match those against its own view.

Again, it's all a question about what is cheaper: a single request to
YARN that results in a large reply, parts of which Livy will ignore
because it's not interested in the data, or hundreds of small requests
to YARN polling specific applications?

> 1. How you express this "current" in a query to YARN? I think you have to
> use ApplicationID (maybe there are some other ways) in a query
>
> 2. The problem is that I didn't see such an API to make such a "big call"
> by passing in all applications's IDs


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

I am using your words *current*. What's the definition of "current" in
livy? I think that's all application which still keep some records in the
livy's process's memory space

So:

1. How you express this "current" in a query to YARN? I think you have to
use ApplicationID (maybe there are some other ways) in a query

2. The problem is that I didn't see such an API to make such a "big call"
by passing in all applications's IDs


Best,

Nan



On Wed, Aug 16, 2017 at 12:19 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> On Wed, Aug 16, 2017 at 12:02 PM, Nan Zhu <zh...@gmail.com> wrote:
> > Then which API you would use for *current* Apps? I think you have to
> define
> > *current* with applicationIds? If that's true, you have to call
> > https://hadoop.apache.org/docs/r2.7.0/api/src-html/org/
> apache/hadoop/yarn/client/api/YarnClient.html#line.181
>
> What do you mean by "current"?
>
> Both the API you linked to and the API I linked to give you "current"
> apps. The one you linked to gives you all "current" apps regardless of
> state. The one I linked to allows you to define which states you're
> interested in. So if you're interested in transitions from RUNNING to
> FAILED, for example, you need to monitor all apps with both states
> RUNNING and FAILED, which that API allows you to do.
>
> There's no need to make N requests as you mentioned.
>
> The question is whether it's cheaper to make a single large request to
> YARN or N small requests. If you are monitoring 4 or 5 applications it
> probably doesn't matter, but if you're monitoring 1000 applications
> that are starting up concurrently, I have a feeling that getting all
> of that information in a single call will be easier on YARN.
>
>
> > If I didn't miss anything, there is no API to pass in a list of app ids,
> as
> > a result, you have to fire N requests (N is the number of current apps)
> to
> > YARN
> >
> > Then the solution becomes using a single thread to fire N request instead
> > of using M threads to fire N requests (ideally M << N)
> >
> >
> >
> >
> > On Wed, Aug 16, 2017 at 11:41 AM, Marcelo Vanzin <va...@cloudera.com>
> > wrote:
> >
> >> On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu <zh...@gmail.com>
> wrote:
> >> > Yes, I know there is such an API, what I don't understand is what I
> >> should
> >> > pass in the filtering API you mentioned, say we query YARN for every 5
> >> > tickets
> >> >
> >> > 0: Query and get App A is running
> >> >
> >> > 4: App A is done
> >> >
> >> > 5: Query...so what I should fill as filtering parameters at 5 get
> capture
> >> > the changes of App A's state?
> >>
> >> You don't query for app state *changes*. You query for the current app
> >> state, and compare against what you have, and then you can detect
> >> changes that way. The trick is how to filter to get the information
> >> you want, so you limit how much data you request from YARN.
> >>
> >> I'm not aware of any YARN API to query for state changes like that. So
> >> even in the individual request case, you'd have to get app A's state,
> >> and update the Livy handle if the state has changed from what was
> >> previously know.
> >>
> >> That's most probably why Meisam's PR only filters by app type. If
> >> there are further filters than can be applied, then great, but you
> >> still need logic in Livy to detect the state changes you want.
> >>
> >> > If you look at Meisam's PR, they can only filter based on appType
> >> > https://github.com/apache/incubator-livy/pull/36/files#diff-
> >> a3f879755cfe10a678cc08ddbe60a4d3R75
> >>
> >>
> >> --
> >> Marcelo
> >>
>
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 12:02 PM, Nan Zhu <zh...@gmail.com> wrote:
> Then which API you would use for *current* Apps? I think you have to define
> *current* with applicationIds? If that's true, you have to call
> https://hadoop.apache.org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/api/YarnClient.html#line.181

What do you mean by "current"?

Both the API you linked to and the API I linked to give you "current"
apps. The one you linked to gives you all "current" apps regardless of
state. The one I linked to allows you to define which states you're
interested in. So if you're interested in transitions from RUNNING to
FAILED, for example, you need to monitor all apps with both states
RUNNING and FAILED, which that API allows you to do.

There's no need to make N requests as you mentioned.

The question is whether it's cheaper to make a single large request to
YARN or N small requests. If you are monitoring 4 or 5 applications it
probably doesn't matter, but if you're monitoring 1000 applications
that are starting up concurrently, I have a feeling that getting all
of that information in a single call will be easier on YARN.


> If I didn't miss anything, there is no API to pass in a list of app ids, as
> a result, you have to fire N requests (N is the number of current apps) to
> YARN
>
> Then the solution becomes using a single thread to fire N request instead
> of using M threads to fire N requests (ideally M << N)
>
>
>
>
> On Wed, Aug 16, 2017 at 11:41 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu <zh...@gmail.com> wrote:
>> > Yes, I know there is such an API, what I don't understand is what I
>> should
>> > pass in the filtering API you mentioned, say we query YARN for every 5
>> > tickets
>> >
>> > 0: Query and get App A is running
>> >
>> > 4: App A is done
>> >
>> > 5: Query...so what I should fill as filtering parameters at 5 get capture
>> > the changes of App A's state?
>>
>> You don't query for app state *changes*. You query for the current app
>> state, and compare against what you have, and then you can detect
>> changes that way. The trick is how to filter to get the information
>> you want, so you limit how much data you request from YARN.
>>
>> I'm not aware of any YARN API to query for state changes like that. So
>> even in the individual request case, you'd have to get app A's state,
>> and update the Livy handle if the state has changed from what was
>> previously know.
>>
>> That's most probably why Meisam's PR only filters by app type. If
>> there are further filters than can be applied, then great, but you
>> still need logic in Livy to detect the state changes you want.
>>
>> > If you look at Meisam's PR, they can only filter based on appType
>> > https://github.com/apache/incubator-livy/pull/36/files#diff-
>> a3f879755cfe10a678cc08ddbe60a4d3R75
>>
>>
>> --
>> Marcelo
>>



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Then which API you would use for *current* Apps? I think you have to define
*current* with applicationIds? If that's true, you have to call
https://hadoop.apache.org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/api/YarnClient.html#line.181
,

If I didn't miss anything, there is no API to pass in a list of app ids, as
a result, you have to fire N requests (N is the number of current apps) to
YARN

Then the solution becomes using a single thread to fire N request instead
of using M threads to fire N requests (ideally M << N)




On Wed, Aug 16, 2017 at 11:41 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu <zh...@gmail.com> wrote:
> > Yes, I know there is such an API, what I don't understand is what I
> should
> > pass in the filtering API you mentioned, say we query YARN for every 5
> > tickets
> >
> > 0: Query and get App A is running
> >
> > 4: App A is done
> >
> > 5: Query...so what I should fill as filtering parameters at 5 get capture
> > the changes of App A's state?
>
> You don't query for app state *changes*. You query for the current app
> state, and compare against what you have, and then you can detect
> changes that way. The trick is how to filter to get the information
> you want, so you limit how much data you request from YARN.
>
> I'm not aware of any YARN API to query for state changes like that. So
> even in the individual request case, you'd have to get app A's state,
> and update the Livy handle if the state has changed from what was
> previously know.
>
> That's most probably why Meisam's PR only filters by app type. If
> there are further filters than can be applied, then great, but you
> still need logic in Livy to detect the state changes you want.
>
> > If you look at Meisam's PR, they can only filter based on appType
> > https://github.com/apache/incubator-livy/pull/36/files#diff-
> a3f879755cfe10a678cc08ddbe60a4d3R75
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 11:34 AM, Nan Zhu <zh...@gmail.com> wrote:
> Yes, I know there is such an API, what I don't understand is what I should
> pass in the filtering API you mentioned, say we query YARN for every 5
> tickets
>
> 0: Query and get App A is running
>
> 4: App A is done
>
> 5: Query...so what I should fill as filtering parameters at 5 get capture
> the changes of App A's state?

You don't query for app state *changes*. You query for the current app
state, and compare against what you have, and then you can detect
changes that way. The trick is how to filter to get the information
you want, so you limit how much data you request from YARN.

I'm not aware of any YARN API to query for state changes like that. So
even in the individual request case, you'd have to get app A's state,
and update the Livy handle if the state has changed from what was
previously know.

That's most probably why Meisam's PR only filters by app type. If
there are further filters than can be applied, then great, but you
still need logic in Livy to detect the state changes you want.

> If you look at Meisam's PR, they can only filter based on appType
> https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Yes, I know there is such an API, what I don't understand is what I should
pass in the filtering API you mentioned, say we query YARN for every 5
tickets

0: Query and get App A is running

4: App A is done

5: Query...so what I should fill as filtering parameters at 5 get capture
the changes of App A's state?

If you look at Meisam's PR, they can only filter based on appType
https://github.com/apache/incubator-livy/pull/36/files#diff-a3f879755cfe10a678cc08ddbe60a4d3R75

I guess it's due to the same reason

Best,

Nan


On Wed, Aug 16, 2017 at 11:28 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> On Wed, Aug 16, 2017 at 11:17 AM, Nan Zhu <zh...@gmail.com> wrote:
> > Looks like non-REST API also contains this https://hadoop.apache.
> > org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/
> > api/YarnClient.html#line.225
> >
> > my concern which was skipped in your last email (again) is that, how many
> > app states we want to fetch through this API. What I can see is we cannot
> > filter applications since application state can change between two polls,
> > any thoughts?
>
> I didn't skip it. I'm intentionally keeping the discussion high level
> because there's no code here to compare. It's purely a "multiple
> requests for single app state" vs. "single request for multiple
> applications' statuses" discussion.
>
> The bulk API I suggested you to investigate should be able to support
> enough filtering so that Livy only gets the information it needs
> (maybe with a little extra noise). It should't get every single YARN
> application ever run, for example.
>
> This method is more what I was thinking of:
>
> 287  public abstract List<ApplicationReport> getApplications(
> 288      Set<String> applicationTypes,
> 289      EnumSet<YarnApplicationState> applicationStates) throws
> YarnException,
> 290      IOException;
>
> Lets you query apps with a given type and multiple states that you're
> interested in. It's not optimal (doesn't let you filter by tags, for
> example), but it's better than getting all apps. Maybe that's now
> enough either, but you're proposing the changes, so please explain why
> that is not enough instead of just throwing the question back at me.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 11:17 AM, Nan Zhu <zh...@gmail.com> wrote:
> Looks like non-REST API also contains this https://hadoop.apache.
> org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/
> api/YarnClient.html#line.225
>
> my concern which was skipped in your last email (again) is that, how many
> app states we want to fetch through this API. What I can see is we cannot
> filter applications since application state can change between two polls,
> any thoughts?

I didn't skip it. I'm intentionally keeping the discussion high level
because there's no code here to compare. It's purely a "multiple
requests for single app state" vs. "single request for multiple
applications' statuses" discussion.

The bulk API I suggested you to investigate should be able to support
enough filtering so that Livy only gets the information it needs
(maybe with a little extra noise). It should't get every single YARN
application ever run, for example.

This method is more what I was thinking of:

287  public abstract List<ApplicationReport> getApplications(
288      Set<String> applicationTypes,
289      EnumSet<YarnApplicationState> applicationStates) throws YarnException,
290      IOException;

Lets you query apps with a given type and multiple states that you're
interested in. It's not optimal (doesn't let you filter by tags, for
example), but it's better than getting all apps. Maybe that's now
enough either, but you're proposing the changes, so please explain why
that is not enough instead of just throwing the question back at me.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

> The JDK has many implementations of concurrent lists and maps. You don't
need to write anything. The code to deal with thread pool vs. the
alternative approach would be different, yes, but you make it sound like
you'd have to implement some really complicated data structure when that is
definitely not the case.

I am not saying that it is complicated, I am saying that, since we can
avoid thread synchronization with the other pros which I have described and
also the cons like more dependencies introduced, which side we are more
interested

> Please don't go down this path. I'm not mixing topic. I made suggestions
that you seemed to not be willing to address, or misunderstood.

I am addressing your suggestions, by asking you what's your concrete ideas
about your suggestion, e.g. how to handle A,B,C cases,

It looks like what you expect is I accept it without any asking


> I didn't blame anyone. I'm trying to get you to address my suggestion and
trying to understand some of your replies, which really were a bit puzzling
to me.

>  If the non-REST API doesn't have such API, then great, you addressed my
suggestion. I don't think it's worth it to rewrite the whole YARN handling
code to use the REST path here. That wasn't hard, was it.


Looks like non-REST API also contains this https://hadoop.apache.
org/docs/r2.7.0/api/src-html/org/apache/hadoop/yarn/client/
api/YarnClient.html#line.225

my concern which was skipped in your last email (again) is that, how many
app states we want to fetch through this API. What I can see is we cannot
filter applications since application state can change between two polls,
any thoughts?




On Wed, Aug 16, 2017 at 10:36 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> On Wed, Aug 16, 2017 at 10:31 AM, Nan Zhu <zh...@gmail.com> wrote:
> >> In the first case your thread pool is the "shared data structure", in
> the
> > second case this map of handles is the "shared data structure", so I
> don't
> > understand why you think there is any difference.
> >
> > I do not understand why there is no difference
> >
> > In your words, when the thread pool is the "shared data structure", we do
> > not need to involve any synchronization when applying CRUD to the
> handles.
> > If you shared "some map or list" between servlet threads and monitoring
> > thread, you have to handle synchronization of this "map or list".
>
> The JDK has many implementations of concurrent lists and maps. You
> don't need to write anything. The code to deal with thread pool vs.
> the alternative approach would be different, yes, but you make it
> sound like you'd have to implement some really complicated data
> structure when that is definitely not the case.
>
> > You mixed two topics again
>
> Please don't go down this path. I'm not mixing topic. I made
> suggestions that you seemed to not be willing to address, or
> misunderstood.
>
> > Topic 2 - How we discuss: I am surprised that you blamed my way to
> discuss
>
> I didn't blame anyone. I'm trying to get you to address my suggestion
> and trying to understand some of your replies, which really were a bit
> puzzling to me.
>
> > OK, I found it,
> > https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/
> ResourceManagerRest.html#Cluster_Applications_API,
> > which is a restful API,
> >
> > two concerns
> >
> > 1. change Livy's current impl is based on YarnClient not restful API, are
> > we going to change it?
>
> If the non-REST API doesn't have such API, then great, you addressed
> my suggestion. I don't think it's worth it to rewrite the whole YARN
> handling code to use the REST path here.
>
> That wasn't hard, was it.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 10:31 AM, Nan Zhu <zh...@gmail.com> wrote:
>> In the first case your thread pool is the "shared data structure", in the
> second case this map of handles is the "shared data structure", so I don't
> understand why you think there is any difference.
>
> I do not understand why there is no difference
>
> In your words, when the thread pool is the "shared data structure", we do
> not need to involve any synchronization when applying CRUD to the handles.
> If you shared "some map or list" between servlet threads and monitoring
> thread, you have to handle synchronization of this "map or list".

The JDK has many implementations of concurrent lists and maps. You
don't need to write anything. The code to deal with thread pool vs.
the alternative approach would be different, yes, but you make it
sound like you'd have to implement some really complicated data
structure when that is definitely not the case.

> You mixed two topics again

Please don't go down this path. I'm not mixing topic. I made
suggestions that you seemed to not be willing to address, or
misunderstood.

> Topic 2 - How we discuss: I am surprised that you blamed my way to discuss

I didn't blame anyone. I'm trying to get you to address my suggestion
and trying to understand some of your replies, which really were a bit
puzzling to me.

> OK, I found it,
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API,
> which is a restful API,
>
> two concerns
>
> 1. change Livy's current impl is based on YarnClient not restful API, are
> we going to change it?

If the non-REST API doesn't have such API, then great, you addressed
my suggestion. I don't think it's worth it to rewrite the whole YARN
handling code to use the REST path here.

That wasn't hard, was it.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

> I really don't understand what you mean. You need somewhere to keep the
application handles you're monitoring regarding of the solution. The code
making the YARN request needs to somehow update those handles. Whether
there's a task per handle that is submitted to a thread pool, or some map
or list tracking all available handles that are then updated by the single
thread talking to YARN, it doesn't matter.

> In the first case your thread pool is the "shared data structure", in the
second case this map of handles is the "shared data structure", so I don't
understand why you think there is any difference.

I do not understand why there is no difference

In your words, when the thread pool is the "shared data structure", we do
not need to involve any synchronization when applying CRUD to the handles.
If you shared "some map or list" between servlet threads and monitoring
thread, you have to handle synchronization of this "map or list". And yes,
if you have single monitoring thread, life can be easier (the potential
cons about single thread to handle everything through bulk operation is
another topic in this email)

> I'm proposing a different approach that I'm pretty sure is easier on YARN,
which is a shared service that we should be trying not to unnecessarily
overload. The least I'd expect is for you to consider the suggestion and
actually explain why it wouldn't work, but so far you've just been
deflecting feedback.

> You can, for example, see if such a bulk API exists and reply "I couldn't
find it". I believe it must exist, after all I can go to the RM web UI and
see all applications, and get a list of them from the YARN REST API. But if
it doesn't exist, that would take care of my suggestion.

You mixed two topics again

Topic 1 - what you proposed: I keep trying to discuss about the pros & cons
of single thread model, and I have said it for multiple times, it can make
life easier, but with additional efforts on 1. synchronization over a
map/list, 2. handling of exceptions (due to the dependency among all
running apps), etc.

Topic 2 - How we discuss: I am surprised that you blamed my way to discuss
even before you correct your own attitude. When we are conducting technical
discussion, I believe you are not in a good position to make it health,
e.g. when I said multiple thread can share the same RPC connections to
avoid your concern about multiple tasks will keep opening/closing
connections, you replied "Irrelevant" without any explanation, and also, I
didn't see any concrete evidence from you about why Actor-based solution is
not an option. On the other side, I have show you my concerns on bulk
operations, which are worth more discussions even though until so far I
just received some feedback like "Irrelevant" or "how about datacenter is
down"


> "I would investigate" is a suggestion that you investigate that as part
of proposing your change. It's not me saying that I'll do it myself (that
would be "I will investigate").

OK, I found it,
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API,
which is a restful API,

two concerns

1. change Livy's current impl is based on YarnClient not restful API, are
we going to change it?

2. being different with RM UI which can only fetch 20 applications for
every page, we need to fetch all applications (since it looks like we
cannot do "not match" against application state through this API)


> I'm expecting that errors be handled regardless of the situation. If YARN
returns an error to you, regardless of whether it was a request for a
single application status or for a bunch of them, your code needs to handle
it somehow. The handling will most probably be the same in both cases
(retry), and that's my point.

yes, this is one of the possible solutions, the pros is it is simple and
easy to handle, the cons is that it makes whether application A's state is
stale depend on all the other applications, which also needs more
discussions


Again, please make technical discussion as professional as possible



On Wed, Aug 16, 2017 at 9:44 AM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 9:33 AM, Nan Zhu <zh...@gmail.com> wrote:
> >> What I proposed is having a single request to YARN to get all
> applications'
> > statuses, if that's possible. You'd still have multiple application
> handles
> > that are independent of each other. They'd all be updated separately from
> > that one thread talking to YARN. This has nothing to do with a "shared
> data
> > structure". There's no shared data structure here to track application
> > status.
> >
> > You are still avoiding the questions how you make all "application
> handles"
> > accessible to this thread
>
> I really don't understand what you mean. You need somewhere to keep
> the application handles you're monitoring regarding of the solution.
> The code making the YARN request needs to somehow update those
> handles. Whether there's a task per handle that is submitted to a
> thread pool, or some map or list tracking all available handles that
> are then updated by the single thread talking to YARN, it doesn't
> matter.
>
> In the first case your thread pool is the "shared data structure", in
> the second case this map of handles is the "shared data structure", so
> I don't understand why you think there is any difference.
>
> I'm proposing a different approach that I'm pretty sure is easier on
> YARN, which is a shared service that we should be trying not to
> unnecessarily overload. The least I'd expect is for you to consider
> the suggestion and actually explain why it wouldn't work, but so far
> you've just been deflecting feedback.
>
> You can, for example, see if such a bulk API exists and reply "I
> couldn't find it". I believe it must exist, after all I can go to the
> RM web UI and see all applications, and get a list of them from the
> YARN REST API. But if it doesn't exist, that would take care of my
> suggestion.
>
> > "I would investigate whether there's any API in YARN to do a bulk get of
> > running applications with a particular filter;" - from your email
> >
> > If you suggest something, please find evidence to support you
>
> "I would investigate" is a suggestion that you investigate that as
> part of proposing your change. It's not me saying that I'll do it
> myself (that would be "I will investigate").
>
> >> What if YARN goes down? What if your datacenter has a massive power
> > failure? You have to handle errors in any scenario.
> >
> > Again, I am describing one concrete scenario which is always involved in
> > any bulk operation and even we go to bulk direction, you have to handle
> > this. Since you proposed this bulk operation, I am asking you what's your
> > expectation about this.
>
> I'm expecting that errors be handled regardless of the situation. If
> YARN returns an error to you, regardless of whether it was a request
> for a single application status or for a bunch of them, your code
> needs to handle it somehow. The handling will most probably be the
> same in both cases (retry), and that's my point.
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 9:33 AM, Nan Zhu <zh...@gmail.com> wrote:
>> What I proposed is having a single request to YARN to get all applications'
> statuses, if that's possible. You'd still have multiple application handles
> that are independent of each other. They'd all be updated separately from
> that one thread talking to YARN. This has nothing to do with a "shared data
> structure". There's no shared data structure here to track application
> status.
>
> You are still avoiding the questions how you make all "application handles"
> accessible to this thread

I really don't understand what you mean. You need somewhere to keep
the application handles you're monitoring regarding of the solution.
The code making the YARN request needs to somehow update those
handles. Whether there's a task per handle that is submitted to a
thread pool, or some map or list tracking all available handles that
are then updated by the single thread talking to YARN, it doesn't
matter.

In the first case your thread pool is the "shared data structure", in
the second case this map of handles is the "shared data structure", so
I don't understand why you think there is any difference.

I'm proposing a different approach that I'm pretty sure is easier on
YARN, which is a shared service that we should be trying not to
unnecessarily overload. The least I'd expect is for you to consider
the suggestion and actually explain why it wouldn't work, but so far
you've just been deflecting feedback.

You can, for example, see if such a bulk API exists and reply "I
couldn't find it". I believe it must exist, after all I can go to the
RM web UI and see all applications, and get a list of them from the
YARN REST API. But if it doesn't exist, that would take care of my
suggestion.

> "I would investigate whether there's any API in YARN to do a bulk get of
> running applications with a particular filter;" - from your email
>
> If you suggest something, please find evidence to support you

"I would investigate" is a suggestion that you investigate that as
part of proposing your change. It's not me saying that I'll do it
myself (that would be "I will investigate").

>> What if YARN goes down? What if your datacenter has a massive power
> failure? You have to handle errors in any scenario.
>
> Again, I am describing one concrete scenario which is always involved in
> any bulk operation and even we go to bulk direction, you have to handle
> this. Since you proposed this bulk operation, I am asking you what's your
> expectation about this.

I'm expecting that errors be handled regardless of the situation. If
YARN returns an error to you, regardless of whether it was a request
for a single application status or for a bunch of them, your code
needs to handle it somehow. The handling will most probably be the
same in both cases (retry), and that's my point.

-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

> What I proposed is having a single request to YARN to get all applications'
statuses, if that's possible. You'd still have multiple application handles
that are independent of each other. They'd all be updated separately from
that one thread talking to YARN. This has nothing to do with a "shared data
structure". There's no shared data structure here to track application
status.

You are still avoiding the questions how you make all "application handles"
accessible to this thread

Please go with direct discussion

> No, but I suggested that you look whether that exists since I think that's
a better solution both from YARN and Livy's perspectives, since it requires
less resources. It should at least be mentioned as an alternative in your
mini-spec and, if it doesn't work for whatever reason, deserves an
explanation.

"I would investigate whether there's any API in YARN to do a bulk get of
running applications with a particular filter;" - from your email

If you suggest something, please find evidence to support you

> Irrelevant.

Please go with direct discussion

> What if YARN goes down? What if your datacenter has a massive power
failure? You have to handle errors in any scenario.

Again, I am describing one concrete scenario which is always involved in
any bulk operation and even we go to bulk direction, you have to handle
this. Since you proposed this bulk operation, I am asking you what's your
expectation about this. But you are throwing some imaginations without any
values

Please go with direct discussion




On Wed, Aug 16, 2017 at 9:11 AM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu <zh...@gmail.com> wrote:
> >> I'm not really sure what you're talking about here, since I did not
> > suggest a "shared data structure", and I'm not really sure what that
> > means in this context.
> >
> > What you claimed is just monitoring/updating the state with a single
> thread
> > *given* all applications have been there.
>
> What I proposed is having a single request to YARN to get all
> applications' statuses, if that's possible. You'd still have multiple
> application handles that are independent of each other. They'd all be
> updated separately from that one thread talking to YARN.
>
> This has nothing to do with a "shared data structure". There's no
> shared data structure here to track application status.
>
> >> Yes. While there are applications that need monitoring, you poll YARN
> > at a constant frequency. Basically what would be done by multiple
> > threads, but there's a single one.
> >
> > Did you find the bulk API?
>
> No, but I suggested that you look whether that exists since I think
> that's a better solution both from YARN and Livy's perspectives, since
> it requires less resources. It should at least be mentioned as an
> alternative in your mini-spec and, if it doesn't work for whatever
> reason, deserves an explanation.
>
> >> Why not. The expensive part is not parsing results, I'll bet, but
> > having a whole bunch of different tasks opening and closing YARN
> > connections.
> >
> > First, YARNClient is thread safe and can be shared by multiple
> threads....
>
> Irrelevant.
>
> > Second, If I have 1000 applications, what's your expectation to the
> > following cases
> >
> > 1. YARN processed request for 999 and failed on the last one for some
> reason
> >
> > 2. Livy received 999 well-formatted response but get 1 malformed response
>
> What if YARN goes down? What if your datacenter has a massive power
> failure?
>
> You have to handle errors in any scenario.
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Aug 16, 2017 at 9:06 AM, Nan Zhu <zh...@gmail.com> wrote:
>> I'm not really sure what you're talking about here, since I did not
> suggest a "shared data structure", and I'm not really sure what that
> means in this context.
>
> What you claimed is just monitoring/updating the state with a single thread
> *given* all applications have been there.

What I proposed is having a single request to YARN to get all
applications' statuses, if that's possible. You'd still have multiple
application handles that are independent of each other. They'd all be
updated separately from that one thread talking to YARN.

This has nothing to do with a "shared data structure". There's no
shared data structure here to track application status.

>> Yes. While there are applications that need monitoring, you poll YARN
> at a constant frequency. Basically what would be done by multiple
> threads, but there's a single one.
>
> Did you find the bulk API?

No, but I suggested that you look whether that exists since I think
that's a better solution both from YARN and Livy's perspectives, since
it requires less resources. It should at least be mentioned as an
alternative in your mini-spec and, if it doesn't work for whatever
reason, deserves an explanation.

>> Why not. The expensive part is not parsing results, I'll bet, but
> having a whole bunch of different tasks opening and closing YARN
> connections.
>
> First, YARNClient is thread safe and can be shared by multiple threads....

Irrelevant.

> Second, If I have 1000 applications, what's your expectation to the
> following cases
>
> 1. YARN processed request for 999 and failed on the last one for some reason
>
> 2. Livy received 999 well-formatted response but get 1 malformed response

What if YARN goes down? What if your datacenter has a massive power failure?

You have to handle errors in any scenario.


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

> I'm not really sure what you're talking about here, since I did not
suggest a "shared data structure", and I'm not really sure what that
means in this context.

What you claimed is just monitoring/updating the state with a single thread
*given* all applications have been there.
 To implement this functionality, you need to make all state of
applications accessible by this thread so that you need some
 data structure storing that (maybe some pointers to Session object)

Then how you put new object to this data structure? through Servlet
directly? then this data structure would be shared by different threads.
make
that thread something like Spark's eventLoop thread? we are seeing the
scalability issues there...

> Yes. While there are applications that need monitoring, you poll YARN
at a constant frequency. Basically what would be done by multiple
threads, but there's a single one.

Did you find the bulk API?

> Why not. The expensive part is not parsing results, I'll bet, but
having a whole bunch of different tasks opening and closing YARN
connections.

First, YARNClient is thread safe and can be shared by multiple threads....


Second, If I have 1000 applications, what's your expectation to the
following cases

1. YARN processed request for 999 and failed on the last one for some reason

2. Livy received 999 well-formatted response but get 1 malformed response




On Tue, Aug 15, 2017 at 5:54 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Tue, Aug 15, 2017 at 2:20 PM, Nan Zhu <zh...@gmail.com> wrote:
> > The key design consideration here is that how you model the state of
> > applications, if in actor, then there will be no synchronization involved
> > and yielding a cleaner design; if in a shared data structure, you will
> have
> > to be careful about coordinating threads here (we actually have a design
> > based on shared data structure and we eventually discard to pursue a
> > cleaner one).
>
> I'm not really sure what you're talking about here, since I did not
> suggest a "shared data structure", and I'm not really sure what that
> means in this context.
>
> > I think bulk API can make life easier comparing to the shared data
> > structure, but it raises up two questions
> >
> > 1. Are we going to update all applications in the uniform pace, even they
> > are submitted in different time?
>
> Yes. While there are applications that need monitoring, you poll YARN
> at a constant frequency. Basically what would be done by multiple
> threads, but there's a single one.
>
> > 2. Are we going to use a single thread for everything, including
> send/recv
> > req/res and parse, etc.
>
> Why not. The expensive part is not parsing results, I'll bet, but
> having a whole bunch of different tasks opening and closing YARN
> connections.
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

On Tue, Aug 15, 2017 at 2:20 PM, Nan Zhu <zh...@gmail.com> wrote:
> The key design consideration here is that how you model the state of
> applications, if in actor, then there will be no synchronization involved
> and yielding a cleaner design; if in a shared data structure, you will have
> to be careful about coordinating threads here (we actually have a design
> based on shared data structure and we eventually discard to pursue a
> cleaner one).

I'm not really sure what you're talking about here, since I did not
suggest a "shared data structure", and I'm not really sure what that
means in this context.

> I think bulk API can make life easier comparing to the shared data
> structure, but it raises up two questions
>
> 1. Are we going to update all applications in the uniform pace, even they
> are submitted in different time?

Yes. While there are applications that need monitoring, you poll YARN
at a constant frequency. Basically what would be done by multiple
threads, but there's a single one.

> 2. Are we going to use a single thread for everything, including send/recv
> req/res and parse, etc.

Why not. The expensive part is not parsing results, I'll bet, but
having a whole bunch of different tasks opening and closing YARN
connections.


-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Hi, Marcelo,

Yes, essentially it is using multiple threads talking with YARN.

The key design consideration here is that how you model the state of
applications, if in actor, then there will be no synchronization involved
and yielding a cleaner design; if in a shared data structure, you will have
to be careful about coordinating threads here (we actually have a design
based on shared data structure and we eventually discard to pursue a
cleaner one).

I think bulk API can make life easier comparing to the shared data
structure, but it raises up two questions

1. Are we going to update all applications in the uniform pace, even they
are submitted in different time?

2. Are we going to use a single thread for everything, including send/recv
req/res and parse, etc.

and we still need to deal with some synchronization,

What do you think?

Best,

Nan






On Tue, Aug 15, 2017 at 11:53 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:

> Hmm, I remember this... it was left as a "todo" item when the app
> monitoring was added.
>
> The document you wrote seems to be a long way of saying you'll have a
> few threads talking to YARN and updating the state of application
> handles in Livy. Is that right?
>
> I would investigate whether there's any API in YARN to do a bulk get
> of running applications with a particular filter; then you could make
> a single call to YARN periodically to get the state of all apps that
> Livy started.
>
>
> On Mon, Aug 14, 2017 at 2:35 PM, Nan Zhu <zh...@gmail.com> wrote:
> > Hi, all
> >
> > In HDInsight, we (Microsoft) use Livy as the Spark job submission
> service.
> > We keep seeing the customers fall into the problem when they submit many
> > concurrent applications to the system, or recover livy from a state with
> > many concurrent applications
> >
> > By looking at the code and the customers' exception stack, we lock down
> the
> > problem to the application monitoring module where a new thread is
> created
> > for each application.
> >
> > To resolve the issue, we propose a actor-based design of application
> > monitoring module and share it here (as new JIRA seems not working
> > yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
> > <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*
> >
> > We are glad to hear feedbacks from the community and improve the design
> > before we start implementing it!
> >
> > Best,
> >
> > Nan
>
>
>
> --
> Marcelo
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Marcelo Vanzin <va...@cloudera.com>]

Hmm, I remember this... it was left as a "todo" item when the app
monitoring was added.

The document you wrote seems to be a long way of saying you'll have a
few threads talking to YARN and updating the state of application
handles in Livy. Is that right?

I would investigate whether there's any API in YARN to do a bulk get
of running applications with a particular filter; then you could make
a single call to YARN periodically to get the state of all apps that
Livy started.


On Mon, Aug 14, 2017 at 2:35 PM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> In HDInsight, we (Microsoft) use Livy as the Spark job submission service.
> We keep seeing the customers fall into the problem when they submit many
> concurrent applications to the system, or recover livy from a state with
> many concurrent applications
>
> By looking at the code and the customers' exception stack, we lock down the
> problem to the application monitoring module where a new thread is created
> for each application.
>
> To resolve the issue, we propose a actor-based design of application
> monitoring module and share it here (as new JIRA seems not working
> yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
> <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*
>
> We are glad to hear feedbacks from the community and improve the design
> before we start implementing it!
>
> Best,
>
> Nan



-- 
Marcelo

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Nan Zhu <zh...@gmail.com>]

Hi, Alex,

Thanks for the reply!

I haven't started implementing it and thought it might be better to get
some feedbacks from the community first

I can start working on it and simultaneously wait for more feedbacks

Best,

Nan

On Mon, Aug 14, 2017 at 4:54 PM, Alex Bozarth <aj...@us.ibm.com> wrote:

> The doc peaks my interest, but I'd have to see some code before deciding
> if its the best direction or not. Hopefully we'll have the new JIRA up by
> end of week so you can submit a JIRA but feel free to open a WIP PR earlier
> if you're ready.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> ------------------------------
> *E-mail:* *ajbozart@us.ibm.com* <aj...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Nan Zhu ---08/14/2017 02:35:22 PM---Hi,
> all In HDInsight, we (Microsoft) use Livy as the Spark job su]Nan Zhu
> ---08/14/2017 02:35:22 PM---Hi, all In HDInsight, we (Microsoft) use Livy
> as the Spark job submission service.
>
> From: Nan Zhu <zh...@gmail.com>
> To: dev@livy.incubator.apache.org
> Date: 08/14/2017 02:35 PM
> Subject: resolve the scalability problem caused by app monitoring in livy
> with an actor-based design
> ------------------------------
>
>
>
> Hi, all
>
> In HDInsight, we (Microsoft) use Livy as the Spark job submission service.
> We keep seeing the customers fall into the problem when they submit many
> concurrent applications to the system, or recover livy from a state with
> many concurrent applications
>
> By looking at the code and the customers' exception stack, we lock down the
> problem to the application monitoring module where a new thread is created
> for each application.
>
> To resolve the issue, we propose a actor-based design of application
> monitoring module and share it here (as new JIRA seems not working
> yet) *https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
> <https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-
> nbTQTdDFXl2XQhXDiwA/edit?usp=sharing>*
>
> We are glad to hear feedbacks from the community and improve the design
> before we start implementing it!
>
> Best,
>
> Nan
>
>
>
>

Re: resolve the scalability problem caused by app monitoring in livy with an actor-based design

[Alex Bozarth <aj...@us.ibm.com>]

The doc peaks my interest, but I'd have to see some code before deciding if
its the best direction or not. Hopefully we'll have the new JIRA up by end
of week so you can submit a JIRA but feel free to open a WIP PR earlier if
you're ready.

                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 E-mail: ajbozart@us.ibm.com                                                         
 GitHub: github.com/ajbozarth                                                        
                                                                   505 Howard Street 
                                                             San Francisco, CA 94105 
                                                                       United States 
                                                                                     








From:	Nan Zhu <zh...@gmail.com>
To:	dev@livy.incubator.apache.org
Date:	08/14/2017 02:35 PM
Subject:	resolve the scalability problem caused by app monitoring in
            livy with an actor-based design



Hi, all

In HDInsight, we (Microsoft) use Livy as the Spark job submission service.
We keep seeing the customers fall into the problem when they submit many
concurrent applications to the system, or recover livy from a state with
many concurrent applications

By looking at the code and the customers' exception stack, we lock down the
problem to the application monitoring module where a new thread is created
for each application.

To resolve the issue, we propose a actor-based design of application
monitoring module and share it here (as new JIRA seems not working
yet)
*https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing

<
https://docs.google.com/document/d/1yDl5_3wPuzyGyFmSOzxRp6P-nbTQTdDFXl2XQhXDiwA/edit?usp=sharing
>*

We are glad to hear feedbacks from the community and improve the design
before we start implementing it!

Best,

Nan