You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by Danushka Menikkumbura <da...@gmail.com> on 2015/01/12 22:19:21 UTC

Spark Examples

Hi devs,

I am trying to understand the behavior of Crunch with Spark. In order to do
that I need to run an example so that I can step through the code. Could
someone point me to any resources that you think would be helpful please?.

Thanks,
Danushka

Re: Spark Examples

Posted by Danushka Menikkumbura <da...@gmail.com>.

Aha!. This is perfect.

Thanks J!

Danushka

On Mon, Jan 12, 2015 at 5:57 PM, Josh Wills <jw...@cloudera.com> wrote:

> Hey Danushka,
>
> The spark branch on my crunch-demo app has a Spark example as well:
>
> https://github.com/jwills/crunch-demo/tree/spark
>
> I/you would need to update the POM to work against version 0.11.0-hadoop2
> and the latest Spark release, either Spark 1.1 or 1.2 should work fine.
>
> J
>
> On Mon, Jan 12, 2015 at 2:27 PM, Danushka Menikkumbura <
> danushka.menikkumbura@gmail.com> wrote:
>
> > Thanks Jinal!
> >
> > On Mon, Jan 12, 2015 at 4:43 PM, Jinal Shah <ji...@gmail.com>
> > wrote:
> >
> > > Hi Danushka,
> > >
> > > The user guide might be a good resource to start with
> > > https://crunch.apache.org/user-guide.html#sparkpipeline
> > >
> > > You can use the Crunch WordCount example:
> > >
> > >
> >
> https://github.com/apache/crunch/blob/master/crunch-archetype/src/main/resources/archetype-resources/src/main/java/WordCount.java
> > >
> > > These are some blogs you can use to learn
> > > http://crunch-tutor.blogspot.com/2014/12/crunch-dofn-example.html
> > > http://engineering.cerner.com/blog/scaling-people-with-apache-crunch/
> > >
> > >
> > > Thanks
> > > Jinal
> > >
> > > On Mon, Jan 12, 2015 at 1:19 PM, Danushka Menikkumbura <
> > > danushka.menikkumbura@gmail.com> wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I am trying to understand the behavior of Crunch with Spark. In order
> > to
> > > do
> > > > that I need to run an example so that I can step through the code.
> > Could
> > > > someone point me to any resources that you think would be helpful
> > > please?.
> > > >
> > > > Thanks,
> > > > Danushka
> > > >
> > >
> >
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Re: Spark Examples

Posted by Josh Wills <jw...@cloudera.com>.

Hey Danushka,

The spark branch on my crunch-demo app has a Spark example as well:

https://github.com/jwills/crunch-demo/tree/spark

I/you would need to update the POM to work against version 0.11.0-hadoop2
and the latest Spark release, either Spark 1.1 or 1.2 should work fine.

J

On Mon, Jan 12, 2015 at 2:27 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:

> Thanks Jinal!
>
> On Mon, Jan 12, 2015 at 4:43 PM, Jinal Shah <ji...@gmail.com>
> wrote:
>
> > Hi Danushka,
> >
> > The user guide might be a good resource to start with
> > https://crunch.apache.org/user-guide.html#sparkpipeline
> >
> > You can use the Crunch WordCount example:
> >
> >
> https://github.com/apache/crunch/blob/master/crunch-archetype/src/main/resources/archetype-resources/src/main/java/WordCount.java
> >
> > These are some blogs you can use to learn
> > http://crunch-tutor.blogspot.com/2014/12/crunch-dofn-example.html
> > http://engineering.cerner.com/blog/scaling-people-with-apache-crunch/
> >
> >
> > Thanks
> > Jinal
> >
> > On Mon, Jan 12, 2015 at 1:19 PM, Danushka Menikkumbura <
> > danushka.menikkumbura@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > I am trying to understand the behavior of Crunch with Spark. In order
> to
> > do
> > > that I need to run an example so that I can step through the code.
> Could
> > > someone point me to any resources that you think would be helpful
> > please?.
> > >
> > > Thanks,
> > > Danushka
> > >
> >
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Re: Spark Examples

Posted by Danushka Menikkumbura <da...@gmail.com>.

Thanks Jinal!

On Mon, Jan 12, 2015 at 4:43 PM, Jinal Shah <ji...@gmail.com> wrote:

> Hi Danushka,
>
> The user guide might be a good resource to start with
> https://crunch.apache.org/user-guide.html#sparkpipeline
>
> You can use the Crunch WordCount example:
>
> https://github.com/apache/crunch/blob/master/crunch-archetype/src/main/resources/archetype-resources/src/main/java/WordCount.java
>
> These are some blogs you can use to learn
> http://crunch-tutor.blogspot.com/2014/12/crunch-dofn-example.html
> http://engineering.cerner.com/blog/scaling-people-with-apache-crunch/
>
>
> Thanks
> Jinal
>
> On Mon, Jan 12, 2015 at 1:19 PM, Danushka Menikkumbura <
> danushka.menikkumbura@gmail.com> wrote:
>
> > Hi devs,
> >
> > I am trying to understand the behavior of Crunch with Spark. In order to
> do
> > that I need to run an example so that I can step through the code. Could
> > someone point me to any resources that you think would be helpful
> please?.
> >
> > Thanks,
> > Danushka
> >
>

Re: Spark Examples

Posted by Jinal Shah <ji...@gmail.com>.

Hi Danushka,

The user guide might be a good resource to start with
https://crunch.apache.org/user-guide.html#sparkpipeline

You can use the Crunch WordCount example:
https://github.com/apache/crunch/blob/master/crunch-archetype/src/main/resources/archetype-resources/src/main/java/WordCount.java

These are some blogs you can use to learn
http://crunch-tutor.blogspot.com/2014/12/crunch-dofn-example.html
http://engineering.cerner.com/blog/scaling-people-with-apache-crunch/


Thanks
Jinal

On Mon, Jan 12, 2015 at 1:19 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:

> Hi devs,
>
> I am trying to understand the behavior of Crunch with Spark. In order to do
> that I need to run an example so that I can step through the code. Could
> someone point me to any resources that you think would be helpful please?.
>
> Thanks,
> Danushka
>