You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nemo.apache.org by John Yang <jo...@gmail.com> on 2018/03/23 01:59:16 UTC

New Nemo blog posts!

Hi Nemo community,



2 new blog posts are up!


http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/


Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
job, and harnessing transient resources in datacenters.


Feel free to ask us any questions about the results, and stay tuned for
more exciting updates! :)



Cheers,

John

Re: New Nemo blog posts!

Posted by John Yang <jo...@gmail.com>.

Thanks Hyunsik for your interest!
Yes, more updates are coming soon. Stay tuned. :)

Cheers,
John


On Fri, Mar 23, 2018 at 3:49 PM, Hyunsik Choi <hy...@apache.org> wrote:

> Hi John,
>
> It's really an interesting article for me because I also have a lot of
> interest in shuffle of distributed processing systems. I'm also looking
> forward to reading further blog articles.
>
> Best regards,
> Hyunsik Choi
>
> On Thu, Mar 22, 2018 at 6:59 PM John Yang <jo...@gmail.com> wrote:
>
> > Hi Nemo community,
> >
> >
> >
> > 2 new blog posts are up!
> >
> >
> > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> >
> >
> > Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> > job, and harnessing transient resources in datacenters.
> >
> >
> > Feel free to ask us any questions about the results, and stay tuned for
> > more exciting updates! :)
> >
> >
> >
> > Cheers,
> >
> > John
> >
>

Re: New Nemo blog posts!

Posted by Hyunsik Choi <hy...@apache.org>.

Hi John,

It's really an interesting article for me because I also have a lot of
interest in shuffle of distributed processing systems. I'm also looking
forward to reading further blog articles.

Best regards,
Hyunsik Choi

On Thu, Mar 22, 2018 at 6:59 PM John Yang <jo...@gmail.com> wrote:

> Hi Nemo community,
>
>
>
> 2 new blog posts are up!
>
>
> http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
>
>
> Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> job, and harnessing transient resources in datacenters.
>
>
> Feel free to ask us any questions about the results, and stay tuned for
> more exciting updates! :)
>
>
>
> Cheers,
>
> John
>

Re: New Nemo blog posts!

Posted by Sanha Lee <sa...@gmail.com>.

Thanks Davor!

I'll check it.

Regards,
Sanha


2018년 3월 27일 (화) 오전 3:33, Davor Bonaci <da...@apache.org>님이 작성:

> Nice posts.
>
> FYI only, you may be interested in [1], which was published a few days
> earlier.
>
> Keep doing the good work!
>
> Davor
>
> [1] https://cloud.google.com/blog/big-data/2018/03/joining-a
> nd-shuffling-very-large-datasets-using-cloud-dataflow
>
> On Fri, Mar 23, 2018 at 12:04 AM, John Yang <jo...@gmail.com> wrote:
>
> > Hi Gon,
> >
> > Good catch! It's Spark DSL for the Spark system, and Beam for Nemo.
> > Will update the articles to make that clear.
> >
> > Cheers,
> > John
> >
> >
> > On Fri, Mar 23, 2018 at 3:59 PM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> >
> > > John,
> > >
> > > From the description, it's not clear the programming layer used by Nemo
> > for
> > > the experiments.
> > > It's Beam, not Spark DSL. Right?
> > >
> > > Cheers,
> > > Gon
> > >
> > > On Fri, Mar 23, 2018 at 11:35 AM, John Yang <jo...@gmail.com>
> wrote:
> > >
> > > > Thanks Gon!
> > > > Updated as per your suggestion. :)
> > > >
> > > > Cheers,
> > > > John
> > > >
> > > > On Fri, Mar 23, 2018 at 11:18 AM, Byung-Gon Chun <bg...@gmail.com>
> > > wrote:
> > > >
> > > > > John, thanks for posting the blogs!
> > > > > The results look great!
> > > > >
> > > > > I have a suggestion. In the second blog, it'd be great if you'd
> > > highlight
> > > > > the numbers and talk about a speedup (120/18).
> > > > >
> > > > > Cheers,
> > > > > Gon
> > > > >
> > > > >
> > > > > On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Nemo community,
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2 new blog posts are up!
> > > > > >
> > > > > >
> > > > > > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> > > > > >
> > > > > >
> > > > > > Come and check out how Nemo outperforms Spark in handling a 2TB
> > > > MapReduce
> > > > > > job, and harnessing transient resources in datacenters.
> > > > > >
> > > > > >
> > > > > > Feel free to ask us any questions about the results, and stay
> tuned
> > > for
> > > > > > more exciting updates! :)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > John
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Byung-Gon Chun
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
>

Re: New Nemo blog posts!

Posted by Davor Bonaci <da...@apache.org>.

Nice posts.

FYI only, you may be interested in [1], which was published a few days
earlier.

Keep doing the good work!

Davor

[1] https://cloud.google.com/blog/big-data/2018/03/joining-a
nd-shuffling-very-large-datasets-using-cloud-dataflow

On Fri, Mar 23, 2018 at 12:04 AM, John Yang <jo...@gmail.com> wrote:

> Hi Gon,
>
> Good catch! It's Spark DSL for the Spark system, and Beam for Nemo.
> Will update the articles to make that clear.
>
> Cheers,
> John
>
>
> On Fri, Mar 23, 2018 at 3:59 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> > John,
> >
> > From the description, it's not clear the programming layer used by Nemo
> for
> > the experiments.
> > It's Beam, not Spark DSL. Right?
> >
> > Cheers,
> > Gon
> >
> > On Fri, Mar 23, 2018 at 11:35 AM, John Yang <jo...@gmail.com> wrote:
> >
> > > Thanks Gon!
> > > Updated as per your suggestion. :)
> > >
> > > Cheers,
> > > John
> > >
> > > On Fri, Mar 23, 2018 at 11:18 AM, Byung-Gon Chun <bg...@gmail.com>
> > wrote:
> > >
> > > > John, thanks for posting the blogs!
> > > > The results look great!
> > > >
> > > > I have a suggestion. In the second blog, it'd be great if you'd
> > highlight
> > > > the numbers and talk about a speedup (120/18).
> > > >
> > > > Cheers,
> > > > Gon
> > > >
> > > >
> > > > On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Nemo community,
> > > > >
> > > > >
> > > > >
> > > > > 2 new blog posts are up!
> > > > >
> > > > >
> > > > > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> > > > >
> > > > >
> > > > > Come and check out how Nemo outperforms Spark in handling a 2TB
> > > MapReduce
> > > > > job, and harnessing transient resources in datacenters.
> > > > >
> > > > >
> > > > > Feel free to ask us any questions about the results, and stay tuned
> > for
> > > > > more exciting updates! :)
> > > > >
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > John
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Byung-Gon Chun
> > > >
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>

Re: New Nemo blog posts!

Posted by John Yang <jo...@gmail.com>.

Hi Gon,

Good catch! It's Spark DSL for the Spark system, and Beam for Nemo.
Will update the articles to make that clear.

Cheers,
John


On Fri, Mar 23, 2018 at 3:59 PM, Byung-Gon Chun <bg...@gmail.com> wrote:

> John,
>
> From the description, it's not clear the programming layer used by Nemo for
> the experiments.
> It's Beam, not Spark DSL. Right?
>
> Cheers,
> Gon
>
> On Fri, Mar 23, 2018 at 11:35 AM, John Yang <jo...@gmail.com> wrote:
>
> > Thanks Gon!
> > Updated as per your suggestion. :)
> >
> > Cheers,
> > John
> >
> > On Fri, Mar 23, 2018 at 11:18 AM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> >
> > > John, thanks for posting the blogs!
> > > The results look great!
> > >
> > > I have a suggestion. In the second blog, it'd be great if you'd
> highlight
> > > the numbers and talk about a speedup (120/18).
> > >
> > > Cheers,
> > > Gon
> > >
> > >
> > > On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com>
> wrote:
> > >
> > > > Hi Nemo community,
> > > >
> > > >
> > > >
> > > > 2 new blog posts are up!
> > > >
> > > >
> > > > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> > > >
> > > >
> > > > Come and check out how Nemo outperforms Spark in handling a 2TB
> > MapReduce
> > > > job, and harnessing transient resources in datacenters.
> > > >
> > > >
> > > > Feel free to ask us any questions about the results, and stay tuned
> for
> > > > more exciting updates! :)
> > > >
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > John
> > > >
> > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
>
>
>
> --
> Byung-Gon Chun
>

Re: New Nemo blog posts!

Posted by Byung-Gon Chun <bg...@gmail.com>.

John,

From the description, it's not clear the programming layer used by Nemo for
the experiments.
It's Beam, not Spark DSL. Right?

Cheers,
Gon

On Fri, Mar 23, 2018 at 11:35 AM, John Yang <jo...@gmail.com> wrote:

> Thanks Gon!
> Updated as per your suggestion. :)
>
> Cheers,
> John
>
> On Fri, Mar 23, 2018 at 11:18 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> > John, thanks for posting the blogs!
> > The results look great!
> >
> > I have a suggestion. In the second blog, it'd be great if you'd highlight
> > the numbers and talk about a speedup (120/18).
> >
> > Cheers,
> > Gon
> >
> >
> > On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com> wrote:
> >
> > > Hi Nemo community,
> > >
> > >
> > >
> > > 2 new blog posts are up!
> > >
> > >
> > > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> > >
> > >
> > > Come and check out how Nemo outperforms Spark in handling a 2TB
> MapReduce
> > > job, and harnessing transient resources in datacenters.
> > >
> > >
> > > Feel free to ask us any questions about the results, and stay tuned for
> > > more exciting updates! :)
> > >
> > >
> > >
> > > Cheers,
> > >
> > > John
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>



-- 
Byung-Gon Chun

Re: New Nemo blog posts!

Posted by John Yang <jo...@gmail.com>.

Thanks Gon!
Updated as per your suggestion. :)

Cheers,
John

On Fri, Mar 23, 2018 at 11:18 AM, Byung-Gon Chun <bg...@gmail.com> wrote:

> John, thanks for posting the blogs!
> The results look great!
>
> I have a suggestion. In the second blog, it'd be great if you'd highlight
> the numbers and talk about a speedup (120/18).
>
> Cheers,
> Gon
>
>
> On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com> wrote:
>
> > Hi Nemo community,
> >
> >
> >
> > 2 new blog posts are up!
> >
> >
> > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> >
> >
> > Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> > job, and harnessing transient resources in datacenters.
> >
> >
> > Feel free to ask us any questions about the results, and stay tuned for
> > more exciting updates! :)
> >
> >
> >
> > Cheers,
> >
> > John
> >
>
>
>
> --
> Byung-Gon Chun
>

Re: New Nemo blog posts!

Posted by Byung-Gon Chun <bg...@gmail.com>.

John, thanks for posting the blogs!
The results look great!

I have a suggestion. In the second blog, it'd be great if you'd highlight
the numbers and talk about a speedup (120/18).

Cheers,
Gon

On Fri, Mar 23, 2018 at 10:59 AM, John Yang <jo...@gmail.com> wrote:

> Hi Nemo community,
>
>
>
> 2 new blog posts are up!
>
>
> http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
>
>
> Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> job, and harnessing transient resources in datacenters.
>
>
> Feel free to ask us any questions about the results, and stay tuned for
> more exciting updates! :)
>
>
>
> Cheers,
>
> John
>

-- 
Byung-Gon Chun

Re: New Nemo blog posts!

Posted by John Yang <jo...@gmail.com>.

Thanks JB! Looking forward to your feedback.

Forgot to mention this, but Sanha and Won Wook lead the development of
SailfishPolicy
and PadoPolicy.
Sanha and Won Wook - feel free to add anything and comment too! :)

Cheers,
John

On Fri, Mar 23, 2018 at 3:24 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Awesome !
>
> Thanks !
>
> Let me take a look and I will come back to you.
>
> Regards
> JB
>
> On 03/23/2018 02:59 AM, John Yang wrote:
> > Hi Nemo community,
> >
> >
> >
> > 2 new blog posts are up!
> >
> >
> > http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> >
> >
> > Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> > job, and harnessing transient resources in datacenters.
> >
> >
> > Feel free to ask us any questions about the results, and stay tuned for
> > more exciting updates! :)
> >
> >
> >
> > Cheers,
> >
> > John
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: New Nemo blog posts!

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.

Awesome !

Thanks !

Let me take a look and I will come back to you.

Regards
JB

On 03/23/2018 02:59 AM, John Yang wrote:
> Hi Nemo community,
> 
> 
> 
> 2 new blog posts are up!
> 
> 
> http://nemo.apache.org/blog/2018/03/23/shuffle-on-nemo/
> 
> 
> Come and check out how Nemo outperforms Spark in handling a 2TB MapReduce
> job, and harnessing transient resources in datacenters.
> 
> 
> Feel free to ask us any questions about the results, and stay tuned for
> more exciting updates! :)
> 
> 
> 
> Cheers,
> 
> John
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com