You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Pawel Szczur <pa...@gmail.com> on 2016/06/01 01:01:54 UTC

Could someone share working bootstrap project with Spark runner configured?

Hi,

A Beam can be easily configured to run against Direct, Dataflow and Flink,
but I couldn't get Spark to work.

Here's a repo I've prepared for a bug reproduction, it may serve as init:
https://github.com/orian/cogroup-wrong-grouping

Could someone modify it or share some working example (outside of Beam
repo).

Cheers, Pawel

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Right, but the spark version is with the scope provided. It means you 
have to provide it yourself. Can you check where the spark jar comes 
from with mvn dependency:tree. It's possible that you have spark jar 
files two times.

Can you share your pom.xml ?

Regards
JB

On 06/03/2016 12:36 AM, Pawel Szczur wrote:
> In the example project Ismael linked, the Beam is a dependency:
> https://github.com/iemejia/beam-playground/blob/master/pom.xml#L35
>
> 2016-06-03 0:23 GMT+02:00 Amit Sela <amitsela33@gmail.com
> <ma...@gmail.com>>:
>
>     You could start with the examples in the README
>     <https://github.com/apache/incubator-beam/tree/master/runners/spark>
>     In addition if you want to run a pipeline in the IDE you should
>     either do it from src/test/... or create a new project and add Beam
>     as a dependency.
>
>     On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pawelszczur@gmail.com
>     <ma...@gmail.com>> wrote:
>
>         I don't understand.
>
>         I've tried to run it from IDE. Could you explain how do you run it?
>
>         2016-06-03 0:08 GMT+02:00 Amit Sela <amitsela33@gmail.com
>         <ma...@gmail.com>>:
>
>             Are you testing under package test ?
>
>
>             On Fri, Jun 3, 2016, 01:06 Pawel Szczur
>             <pawelszczur@gmail.com <ma...@gmail.com>> wrote:
>
>                 I've just tested, it throws an exception with nightly Beam:
>                 Exception in thread "main"
>                 java.lang.NoClassDefFoundError:
>                 org/apache/spark/api/java/JavaSparkContext
>
>
>                 2016-06-01 14:23 GMT+02:00 Isma�l Mej�a
>                 <iemejia@gmail.com <ma...@gmail.com>>:
>
>                     You can find my pom.xml file for my beam-playground
>                     project that runs in the spark runner, note that it
>                     uses the daily snapshots, but I will change it for
>                     the released jars once they are on maven central:
>
>                     https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
>                     Regards,
>                     Isma�l
>
>                     On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur
>                     <pawelszczur@gmail.com
>                     <ma...@gmail.com>> wrote:
>
>                         Hi,
>
>                         A Beam can be easily configured to run
>                         against Direct, Dataflow and Flink, but I
>                         couldn't get Spark to work.
>
>                         Here's a repo I've prepared for a bug
>                         reproduction, it may serve as init:
>                         https://github.com/orian/cogroup-wrong-grouping
>
>                         Could someone modify it or share some working
>                         example (outside of Beam repo).
>
>                         Cheers, Pawel
>
>
>
>
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Pawel Szczur <pa...@gmail.com>.
In the example project Ismael linked, the Beam is a dependency:
https://github.com/iemejia/beam-playground/blob/master/pom.xml#L35

2016-06-03 0:23 GMT+02:00 Amit Sela <am...@gmail.com>:

> You could start with the examples in the README
> <https://github.com/apache/incubator-beam/tree/master/runners/spark>
> In addition if you want to run a pipeline in the IDE you should either do
> it from src/test/... or create a new project and add Beam as a dependency.
>
> On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pa...@gmail.com> wrote:
>
>> I don't understand.
>>
>> I've tried to run it from IDE. Could you explain how do you run it?
>>
>> 2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:
>>
>>> Are you testing under package test ?
>>>
>>> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>>>
>>>> I've just tested, it throws an exception with nightly Beam:
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/spark/api/java/JavaSparkContext
>>>>
>>>>
>>>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>>>
>>>>> You can find my pom.xml file for my beam-playground project that runs
>>>>> in the spark runner, note that it uses the daily snapshots, but I will
>>>>> change it for the released jars once they are on maven central:
>>>>>
>>>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>>>
>>>>> Regards,
>>>>> Ismaël
>>>>>
>>>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>>>> Flink, but I couldn't get Spark to work.
>>>>>>
>>>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>>>> init:
>>>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>>>
>>>>>> Could someone modify it or share some working example (outside of
>>>>>> Beam repo).
>>>>>>
>>>>>> Cheers, Pawel
>>>>>>
>>>>>
>>>>>
>>>>
>>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Amit Sela <am...@gmail.com>.
You could start with the examples in the README
<https://github.com/apache/incubator-beam/tree/master/runners/spark>
In addition if you want to run a pipeline in the IDE you should either do
it from src/test/... or create a new project and add Beam as a dependency.

On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pa...@gmail.com> wrote:

> I don't understand.
>
> I've tried to run it from IDE. Could you explain how do you run it?
>
> 2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:
>
>> Are you testing under package test ?
>>
>> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>>
>>> I've just tested, it throws an exception with nightly Beam:
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/spark/api/java/JavaSparkContext
>>>
>>>
>>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>>
>>>> You can find my pom.xml file for my beam-playground project that runs
>>>> in the spark runner, note that it uses the daily snapshots, but I will
>>>> change it for the released jars once they are on maven central:
>>>>
>>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>>
>>>> Regards,
>>>> Ismaël
>>>>
>>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>>> Flink, but I couldn't get Spark to work.
>>>>>
>>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>>> init:
>>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>>
>>>>> Could someone modify it or share some working example (outside of Beam
>>>>> repo).
>>>>>
>>>>> Cheers, Pawel
>>>>>
>>>>
>>>>
>>>
>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Pawel Szczur <pa...@gmail.com>.
I don't understand.

I've tried to run it from IDE. Could you explain how do you run it?

2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:

> Are you testing under package test ?
>
> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>
>> I've just tested, it throws an exception with nightly Beam:
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/spark/api/java/JavaSparkContext
>>
>>
>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>
>>> You can find my pom.xml file for my beam-playground project that runs in
>>> the spark runner, note that it uses the daily snapshots, but I will change
>>> it for the released jars once they are on maven central:
>>>
>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>
>>> Regards,
>>> Ismaël
>>>
>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>> Flink, but I couldn't get Spark to work.
>>>>
>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>> init:
>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>
>>>> Could someone modify it or share some working example (outside of Beam
>>>> repo).
>>>>
>>>> Cheers, Pawel
>>>>
>>>
>>>
>>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Amit Sela <am...@gmail.com>.
Are you testing under package test ?

On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:

> I've just tested, it throws an exception with nightly Beam:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/JavaSparkContext
>
>
> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>
>> You can find my pom.xml file for my beam-playground project that runs in
>> the spark runner, note that it uses the daily snapshots, but I will change
>> it for the released jars once they are on maven central:
>>
>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>
>> Regards,
>> Ismaël
>>
>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> A Beam can be easily configured to run against Direct, Dataflow and
>>> Flink, but I couldn't get Spark to work.
>>>
>>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>>> https://github.com/orian/cogroup-wrong-grouping
>>>
>>> Could someone modify it or share some working example (outside of Beam
>>> repo).
>>>
>>> Cheers, Pawel
>>>
>>
>>
>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Pawel Szczur <pa...@gmail.com>.
I've just tested, it throws an exception with nightly Beam:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/api/java/JavaSparkContext


2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:

> You can find my pom.xml file for my beam-playground project that runs in
> the spark runner, note that it uses the daily snapshots, but I will change
> it for the released jars once they are on maven central:
>
> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
> Regards,
> Ismaël
>
> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> A Beam can be easily configured to run against Direct, Dataflow and
>> Flink, but I couldn't get Spark to work.
>>
>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>> https://github.com/orian/cogroup-wrong-grouping
>>
>> Could someone modify it or share some working example (outside of Beam
>> repo).
>>
>> Cheers, Pawel
>>
>
>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Pawel Szczur <pa...@gmail.com>.
Thanks Ismael, I was looking for something like that.

Amit: yes. It's only hard coded for the purpose of replicating a bug.

2016-06-01 14:38 GMT+02:00 Amit Sela <am...@gmail.com>:

> Did you try changing FlinkPipelineOptions and FlinkPipelineRunner to
> Spark's equivalents ? It looks like it's hard coded to run on Flink.
>
> On Wed, Jun 1, 2016 at 3:23 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You can find my pom.xml file for my beam-playground project that runs in
>> the spark runner, note that it uses the daily snapshots, but I will change
>> it for the released jars once they are on maven central:
>>
>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>
>> Regards,
>> Ismaël
>>
>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> A Beam can be easily configured to run against Direct, Dataflow and
>>> Flink, but I couldn't get Spark to work.
>>>
>>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>>> https://github.com/orian/cogroup-wrong-grouping
>>>
>>> Could someone modify it or share some working example (outside of Beam
>>> repo).
>>>
>>> Cheers, Pawel
>>>
>>
>>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Amit Sela <am...@gmail.com>.
Did you try changing FlinkPipelineOptions and FlinkPipelineRunner to
Spark's equivalents ? It looks like it's hard coded to run on Flink.

On Wed, Jun 1, 2016 at 3:23 PM Ismaël Mejía <ie...@gmail.com> wrote:

> You can find my pom.xml file for my beam-playground project that runs in
> the spark runner, note that it uses the daily snapshots, but I will change
> it for the released jars once they are on maven central:
>
> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
> Regards,
> Ismaël
>
> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> A Beam can be easily configured to run against Direct, Dataflow and
>> Flink, but I couldn't get Spark to work.
>>
>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>> https://github.com/orian/cogroup-wrong-grouping
>>
>> Could someone modify it or share some working example (outside of Beam
>> repo).
>>
>> Cheers, Pawel
>>
>
>

Re: Could someone share working bootstrap project with Spark runner configured?

Posted by Ismaël Mejía <ie...@gmail.com>.
You can find my pom.xml file for my beam-playground project that runs in
the spark runner, note that it uses the daily snapshots, but I will change
it for the released jars once they are on maven central:

https://github.com/iemejia/beam-playground/blob/master/pom.xml

Regards,
Ismaël

On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com> wrote:

> Hi,
>
> A Beam can be easily configured to run against Direct, Dataflow and
> Flink, but I couldn't get Spark to work.
>
> Here's a repo I've prepared for a bug reproduction, it may serve as init:
> https://github.com/orian/cogroup-wrong-grouping
>
> Could someone modify it or share some working example (outside of Beam
> repo).
>
> Cheers, Pawel
>