Nemo @ ApacheCon NA 2019

[John Youngseok Yang <jo...@gmail.com>]

Hi Nemoers,

Just wanted to share my trip to ApacheCon NA 2019. :)
Overall it was great fun. I met and talked with many folks working on a
number of different Apache big data projects.
Several people who attended Wonwook's talk at Beam summit EU already knew
about Nemo. Kudos to Wonwook for the excellent job!

People really liked how Nemo transparently integrates with existing
programming libraries (Beam, Spark), and cluster resource managers (YARN,
Mesos).
This seems to validate the decision we made early on to make Nemo
compatible with other Apache big data projects including REEF.
People also liked the optimizations, and some of them were curious whether
the optimization options can be configured automatically for them.
I suppose this validates the "smart compiler" works some of us are
currently working on.

I did 2 talks at the conference.
One was the main Nemo talk, and the other was a demo of running Nemo on
Google DataProc in a Beam session.
The main talk was primarily about how Nemo optimizes distributed execution.
On the next day, Beam organizers generously let me do a demo talk to
demonstrate how easy it is to spin up new cloud instances, and run Beam
programs using the Nemo runner.

After seeing the demo one of the attendees there suggested that I file a PR
to a GCP repository (
https://github.com/GoogleCloudPlatform/dataproc-initialization-actions) to
automate the installation process, and we can do a blog post on using Nemo
on Google Dataproc.
I'll work on this to make Nemo more accessible to GCP users.

Thank Nemoers for all your work!

Cheers,
John

Re: Nemo @ ApacheCon NA 2019

[Byung-Gon Chun <bg...@gmail.com>]

Hi John,

Thank you for sharing your experience with us!
Great work at ApacheCon!

Making Nemo work on Google Dataproc is a great idea.
If Beam users don't use cloud dataflow, they can definitely try out Nemo as
an open-source runner.

Thanks!
-Gon



On Sat, Sep 21, 2019 at 9:53 AM John Youngseok Yang <jo...@gmail.com>
wrote:

> Hi Nemoers,
>
> Just wanted to share my trip to ApacheCon NA 2019. :)
> Overall it was great fun. I met and talked with many folks working on a
> number of different Apache big data projects.
> Several people who attended Wonwook's talk at Beam summit EU already knew
> about Nemo. Kudos to Wonwook for the excellent job!
>
> People really liked how Nemo transparently integrates with existing
> programming libraries (Beam, Spark), and cluster resource managers (YARN,
> Mesos).
> This seems to validate the decision we made early on to make Nemo
> compatible with other Apache big data projects including REEF.
> People also liked the optimizations, and some of them were curious whether
> the optimization options can be configured automatically for them.
> I suppose this validates the "smart compiler" works some of us are
> currently working on.
>
> I did 2 talks at the conference.
> One was the main Nemo talk, and the other was a demo of running Nemo on
> Google DataProc in a Beam session.
> The main talk was primarily about how Nemo optimizes distributed execution.
> On the next day, Beam organizers generously let me do a demo talk to
> demonstrate how easy it is to spin up new cloud instances, and run Beam
> programs using the Nemo runner.
>
> After seeing the demo one of the attendees there suggested that I file a PR
> to a GCP repository (
> https://github.com/GoogleCloudPlatform/dataproc-initialization-actions) to
> automate the installation process, and we can do a blog post on using Nemo
> on Google Dataproc.
> I'll work on this to make Nemo more accessible to GCP users.
>
> Thank Nemoers for all your work!
>
> Cheers,
> John
>


-- 
Byung-Gon Chun