Introducing Heron on REEF + YARN

[Ashvin A <aa...@gmail.com>]

Hi,

I recently used REEF to enable Heron [1] deployment on YARN clusters. I
wanted to share some information about the work with this community.

Heron is a successor of Apache Storm (stream processing), recently open
sourced by Twitter [2]. Initially Heron supported Aurora and Mesos clusters
only. My goal was to add YARN cluster support. I leverage the REEF
framework to develop a custom Heron scheduler [3] for YARN.

Heron executes user workflows, called topologies [4], to process data
streams. Each topology is a long running service, which would become a long
running YARN job. The topology needs to be restarted whenever a user
updates the topology code. With these initial requirements I tried Slider,
YARN API and REEF for implementing this scheduler.

I found the REEF framework to be addressing the requirements very well.
Particularly the ability to retain container when topology is restarted
significantly reduced down time. The user creates a topology package and it
seemed unnecessary to create another package layer as required by Slider.
REEF's event model for exception and failure management was also very
useful. I was able to get the initial version running quickly.

The YARN scheduler is still evolving. Currently the YARN AM HA is the
biggest missing pease. I have created a user doc [5] for deployment on a
YARN cluster in case you are interested in trying it.

I am very impressed with the REEF framework and am looking forward to
participate in the community.

Thanks,
Ashvin

Disclaimer: I am working at Microsoft

[1] http://twitter.github.io/heron
[2] https://blog.twitter.com/2016/open-sourcing-twitter-heron
[3] http://twitter.github.io/heron/docs/contributors/custom-scheduler/
[4] http://twitter.github.io/heron/docs/concepts/topologies/
[5]
https://github.com/ashvina/heron/blob/yarnSchedulerDoc-947/website/content/docs/operators/deployment/schedulers/yarn.md

Re: Introducing Heron on REEF + YARN

[Markus Weimer <ma...@weimo.de>]

On 2016-06-30 17:43, Ashvin A wrote:
> I recently used REEF to enable Heron [1] deployment on YARN clusters. I
> wanted to share some information about the work with this community.

Awesome, thanks! I appreciate all the positive feedback. I am glad you 
had a good time.

Did you also stumble over things we can do better? Relative newcomers 
are the best resource we have to learn about unintuitive choices we made 
in our code and documentation. In other words: feel free to be blunt. We 
are all friends here :)

Markus

Re: Introducing Heron on REEF + YARN

[Byung-Gon Chun <bg...@gmail.com>]

Ashvin,

This is great. Thank you for sharing your experience!

-Gon

On Fri, Jul 1, 2016 at 9:43 AM, Ashvin A <aa...@gmail.com> wrote:

> Hi,
>
> I recently used REEF to enable Heron [1] deployment on YARN clusters. I
> wanted to share some information about the work with this community.
>
> Heron is a successor of Apache Storm (stream processing), recently open
> sourced by Twitter [2]. Initially Heron supported Aurora and Mesos clusters
> only. My goal was to add YARN cluster support. I leverage the REEF
> framework to develop a custom Heron scheduler [3] for YARN.
>
> Heron executes user workflows, called topologies [4], to process data
> streams. Each topology is a long running service, which would become a long
> running YARN job. The topology needs to be restarted whenever a user
> updates the topology code. With these initial requirements I tried Slider,
> YARN API and REEF for implementing this scheduler.
>
> I found the REEF framework to be addressing the requirements very well.
> Particularly the ability to retain container when topology is restarted
> significantly reduced down time. The user creates a topology package and it
> seemed unnecessary to create another package layer as required by Slider.
> REEF's event model for exception and failure management was also very
> useful. I was able to get the initial version running quickly.
>
> The YARN scheduler is still evolving. Currently the YARN AM HA is the
> biggest missing pease. I have created a user doc [5] for deployment on a
> YARN cluster in case you are interested in trying it.
>
> I am very impressed with the REEF framework and am looking forward to
> participate in the community.
>
> Thanks,
> Ashvin
>
> Disclaimer: I am working at Microsoft
>
> [1] http://twitter.github.io/heron
> [2] https://blog.twitter.com/2016/open-sourcing-twitter-heron
> [3] http://twitter.github.io/heron/docs/contributors/custom-scheduler/
> [4] http://twitter.github.io/heron/docs/concepts/topologies/
> [5]
>
> https://github.com/ashvina/heron/blob/yarnSchedulerDoc-947/website/content/docs/operators/deployment/schedulers/yarn.md
>



-- 
Byung-Gon Chun