R-Storm - Resource Aware Scheduling implementation for Storm

["Boyang(Jerry) Peng" <je...@yahoo-inc.com>]

Hello Everyone,



I recently published a paper in ACM Middleware on the subject of Resource Aware Scheduling in Storm in which I demonstrated how the scheduling algorithms I used can significantly improve performance and stability of Storm.  I have created an implementation for open source based off of a implementation I worked on a Yahoo. The implementation is on my github. The link is provided below:
https://github.com/jerrypeng/storm/tree/opensource_ras

The link to the paper: 
http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf

The paper describes the architecture, algorithms, and API of the Resource Aware Scheduler as well as an evaluation of performance.  The research and initial implementation was done by me while I was a graduate student at University of Illinois, Urbana-champaign.  After I graduated, I joined Yahoo and we came up with a better implementation.
The implementation should work straight out of the box.  If you are running storm in a multi-rack environment, and you want to leverage the network aware portion of the scheduler to improve performance and decrease latency, you can implement your own STORM_NETWORK_TOPOLOGY_PLUGIN that maps the underlying physical network.  The default plugin that comes with the implementation assumes all machines have the same network distance from each other i.e. on the same rack.  Contact me if you need more information on this subject.
Please look at this initial implementation and provide me with some feedback, since we intend to have a pull request up soon and merge it into master.
People that worked on the implementation at Yahoo with me:

Bobby Evans (Yahoo & Storm PMC)Derek Dagit (Yahoo & Storm PMC)Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)Sanket Chintapalli (Yahoo)Reza Fravier (Yahoo & UIUC)
Best,
Boyang Jerry PengSoftware Engineer, Yahoohttp://web.engr.illinois.edu/~bpeng/



   

Re: R-Storm - Resource Aware Scheduling implementation for Storm

[Nithesh Kb <ni...@gmail.com>]

Hi Jerry,
It is very interesting.
Could you please share the recorded session of your demo ?

Regards,
Nithesh
On Aug 20, 2015 1:24 AM, "Boyang(Jerry) Peng" <je...@yahoo-inc.com>
wrote:

> Hello Everyone,
>
>
>
> I recently published a paper in ACM Middleware on the subject of Resource
> Aware Scheduling in Storm in which I demonstrated how the scheduling
> algorithms I used can significantly improve performance and stability of
> Storm.  I have created an implementation for open source based off of a
> implementation I worked on a Yahoo. The implementation is on my github. The
> link is provided below:
>
> https://github.com/jerrypeng/storm/tree/opensource_ras
>
> The link to the paper:
>
> http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
>
> The paper describes the architecture, algorithms, and API of the Resource
> Aware Scheduler as well as an evaluation of performance.  The research and
> initial implementation was done by me while I was a graduate student at
> University of Illinois, Urbana-champaign.  After I graduated, I joined
> Yahoo and we came up with a better implementation.
>
> The implementation should work straight out of the box.  If you are
> running storm in a multi-rack environment, and you want to leverage the
> network aware portion of the scheduler to improve performance and decrease
> latency, you can implement your own STORM_NETWORK_TOPOLOGY_PLUGIN that maps
> the underlying physical network.  The default plugin that comes with the
> implementation assumes all machines have the same network distance from
> each other i.e. on the same rack.  Contact me if you need more information
> on this subject.
>
> *Please look at this initial implementation and provide me with some
> feedback, since we intend to have a pull request up soon and merge it into
> master.*
>
> People that worked on the implementation at Yahoo with me:
>
> Bobby Evans (Yahoo & Storm PMC)
> Derek Dagit (Yahoo & Storm PMC)
> Kyle Nusbaum (Yahoo & Storm PMC)
> Liu Zhuo (Yahoo)
> Sanket Chintapalli (Yahoo)
> Reza Fravier (Yahoo & UIUC)
>
> Best,
>
> Boyang Jerry Peng
> Software Engineer, Yahoo
> http://web.engr.illinois.edu/~bpeng/
>
>
>
>