Question about sending YARN Scheduler Events to Client Applications

[Charlie Wright <ch...@live.ca>]

I am trying to send an event to Spark/Client Applications via the YARN Scheduler and I am not sure how to go about doing so. I have an event type that I have created in SchedulerEventType and have created a corresponding SchedulerEvent. The event is dispatched to the ResourceManager's dispatcher and sent to SchedulerEventDispatcher where it dispatches the event to the Scheduler. After that, I am not sure where to go from there. How can I get events to the client? Is this done through ApplicationReport->YarnApplicationState ? 
Thanks,Charles.
 		 	   		  

RE: Question about sending YARN Scheduler Events to Client Applications

[Charlie Wright <ch...@live.ca>]

Well, what I have been trying to is send a "Cluster Pressure Event" to the Spark Applications running on a cluster to allow for dynamic re-sizing of Spark applications running on the cluster. It is a 4th year project that I am working on for my university program. We have the Spark side implemented where spark applications are set to have a fair share value for the number of executors along with the already implemented max and min values. This way, a spark app will grow until it has it's max and then throttle back to it's fair share if a Cluster Pressure Event is triggered. The motivation behind this, is to allow for interactive jobs to come onto the cluster - so that they don't have to wait for the other jobs to complete. Currently the only way for a application to "shrink down" is if the executors time out for inactivity. We have defined cluster pressure as (pending resources + allocated resources / total resources > 1) meaning that there has been more requests for resources than what is available - so we need to release some. Hopefully, that gives you some insight as to what I am trying to do - I've been trying to figure out how to implement this for sometime, but have been running into various implementation roadblocks. If more information is required, I can provide it.
-Charles. 
> Subject: Re: Question about sending YARN Scheduler Events to Client Applications
> From: vinodkv@apache.org
> Date: Wed, 9 Mar 2016 14:07:03 -0800
> To: yarn-dev@hadoop.apache.org
> 
> The code path you referred to is completely YARN-internal only and so won’t serve any purpose of communication to clients or applications.
> 
> What exactly are you trying to do? If you share more details, may be we can help you in figuring out how to do it.
> 
> Thanks
> +Vinod
> 
> 
> > On Mar 9, 2016, at 12:09 PM, Charlie Wright <ch...@live.ca> wrote:
> > 
> > 
> > 
> > 
> > I am trying to send an event to Spark/Client Applications via the YARN Scheduler and I am not sure how to go about doing so. I have an event type that I have created in SchedulerEventType and have created a corresponding SchedulerEvent. The event is dispatched to the ResourceManager's dispatcher and sent to SchedulerEventDispatcher where it dispatches the event to the Scheduler. After that, I am not sure where to go from there. How can I get events to the client? Is this done through ApplicationReport->YarnApplicationState ? 
> > Thanks,Charles.
> > 		 	   		  
> 
 		 	   		  

Re: Question about sending YARN Scheduler Events to Client Applications

[Vinod Kumar Vavilapalli <vi...@apache.org>]

The code path you referred to is completely YARN-internal only and so won’t serve any purpose of communication to clients or applications.

What exactly are you trying to do? If you share more details, may be we can help you in figuring out how to do it.

Thanks
+Vinod


> On Mar 9, 2016, at 12:09 PM, Charlie Wright <ch...@live.ca> wrote:
> 
> 
> 
> 
> I am trying to send an event to Spark/Client Applications via the YARN Scheduler and I am not sure how to go about doing so. I have an event type that I have created in SchedulerEventType and have created a corresponding SchedulerEvent. The event is dispatched to the ResourceManager's dispatcher and sent to SchedulerEventDispatcher where it dispatches the event to the Scheduler. After that, I am not sure where to go from there. How can I get events to the client? Is this done through ApplicationReport->YarnApplicationState ? 
> Thanks,Charles.
>