You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Pawel Szczur <pa...@gmail.com> on 2016/06/01 01:01:54 UTC
Could someone share working bootstrap project with Spark runner configured?
Hi,
A Beam can be easily configured to run against Direct, Dataflow and Flink,
but I couldn't get Spark to work.
Here's a repo I've prepared for a bug reproduction, it may serve as init:
https://github.com/orian/cogroup-wrong-grouping
Could someone modify it or share some working example (outside of Beam
repo).
Cheers, Pawel
Re: Could someone share working bootstrap project with Spark runner
configured?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Right, but the spark version is with the scope provided. It means you
have to provide it yourself. Can you check where the spark jar comes
from with mvn dependency:tree. It's possible that you have spark jar
files two times.
Can you share your pom.xml ?
Regards
JB
On 06/03/2016 12:36 AM, Pawel Szczur wrote:
> In the example project Ismael linked, the Beam is a dependency:
> https://github.com/iemejia/beam-playground/blob/master/pom.xml#L35
>
> 2016-06-03 0:23 GMT+02:00 Amit Sela <amitsela33@gmail.com
> <ma...@gmail.com>>:
>
> You could start with the examples in the README
> <https://github.com/apache/incubator-beam/tree/master/runners/spark>
> In addition if you want to run a pipeline in the IDE you should
> either do it from src/test/... or create a new project and add Beam
> as a dependency.
>
> On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pawelszczur@gmail.com
> <ma...@gmail.com>> wrote:
>
> I don't understand.
>
> I've tried to run it from IDE. Could you explain how do you run it?
>
> 2016-06-03 0:08 GMT+02:00 Amit Sela <amitsela33@gmail.com
> <ma...@gmail.com>>:
>
> Are you testing under package test ?
>
>
> On Fri, Jun 3, 2016, 01:06 Pawel Szczur
> <pawelszczur@gmail.com <ma...@gmail.com>> wrote:
>
> I've just tested, it throws an exception with nightly Beam:
> Exception in thread "main"
> java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/JavaSparkContext
>
>
> 2016-06-01 14:23 GMT+02:00 Isma�l Mej�a
> <iemejia@gmail.com <ma...@gmail.com>>:
>
> You can find my pom.xml file for my beam-playground
> project that runs in the spark runner, note that it
> uses the daily snapshots, but I will change it for
> the released jars once they are on maven central:
>
> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
> Regards,
> Isma�l
>
> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur
> <pawelszczur@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> A Beam can be easily configured to run
> against Direct, Dataflow and Flink, but I
> couldn't get Spark to work.
>
> Here's a repo I've prepared for a bug
> reproduction, it may serve as init:
> https://github.com/orian/cogroup-wrong-grouping
>
> Could someone modify it or share some working
> example (outside of Beam repo).
>
> Cheers, Pawel
>
>
>
>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Pawel Szczur <pa...@gmail.com>.
In the example project Ismael linked, the Beam is a dependency:
https://github.com/iemejia/beam-playground/blob/master/pom.xml#L35
2016-06-03 0:23 GMT+02:00 Amit Sela <am...@gmail.com>:
> You could start with the examples in the README
> <https://github.com/apache/incubator-beam/tree/master/runners/spark>
> In addition if you want to run a pipeline in the IDE you should either do
> it from src/test/... or create a new project and add Beam as a dependency.
>
> On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pa...@gmail.com> wrote:
>
>> I don't understand.
>>
>> I've tried to run it from IDE. Could you explain how do you run it?
>>
>> 2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:
>>
>>> Are you testing under package test ?
>>>
>>> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>>>
>>>> I've just tested, it throws an exception with nightly Beam:
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/spark/api/java/JavaSparkContext
>>>>
>>>>
>>>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>>>
>>>>> You can find my pom.xml file for my beam-playground project that runs
>>>>> in the spark runner, note that it uses the daily snapshots, but I will
>>>>> change it for the released jars once they are on maven central:
>>>>>
>>>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>>>
>>>>> Regards,
>>>>> Ismaël
>>>>>
>>>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>>>> Flink, but I couldn't get Spark to work.
>>>>>>
>>>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>>>> init:
>>>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>>>
>>>>>> Could someone modify it or share some working example (outside of
>>>>>> Beam repo).
>>>>>>
>>>>>> Cheers, Pawel
>>>>>>
>>>>>
>>>>>
>>>>
>>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Amit Sela <am...@gmail.com>.
You could start with the examples in the README
<https://github.com/apache/incubator-beam/tree/master/runners/spark>
In addition if you want to run a pipeline in the IDE you should either do
it from src/test/... or create a new project and add Beam as a dependency.
On Fri, Jun 3, 2016 at 1:18 AM Pawel Szczur <pa...@gmail.com> wrote:
> I don't understand.
>
> I've tried to run it from IDE. Could you explain how do you run it?
>
> 2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:
>
>> Are you testing under package test ?
>>
>> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>>
>>> I've just tested, it throws an exception with nightly Beam:
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/spark/api/java/JavaSparkContext
>>>
>>>
>>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>>
>>>> You can find my pom.xml file for my beam-playground project that runs
>>>> in the spark runner, note that it uses the daily snapshots, but I will
>>>> change it for the released jars once they are on maven central:
>>>>
>>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>>
>>>> Regards,
>>>> Ismaël
>>>>
>>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>>> Flink, but I couldn't get Spark to work.
>>>>>
>>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>>> init:
>>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>>
>>>>> Could someone modify it or share some working example (outside of Beam
>>>>> repo).
>>>>>
>>>>> Cheers, Pawel
>>>>>
>>>>
>>>>
>>>
>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Pawel Szczur <pa...@gmail.com>.
I don't understand.
I've tried to run it from IDE. Could you explain how do you run it?
2016-06-03 0:08 GMT+02:00 Amit Sela <am...@gmail.com>:
> Are you testing under package test ?
>
> On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
>
>> I've just tested, it throws an exception with nightly Beam:
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/spark/api/java/JavaSparkContext
>>
>>
>> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>>
>>> You can find my pom.xml file for my beam-playground project that runs in
>>> the spark runner, note that it uses the daily snapshots, but I will change
>>> it for the released jars once they are on maven central:
>>>
>>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>>
>>> Regards,
>>> Ismaël
>>>
>>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> A Beam can be easily configured to run against Direct, Dataflow and
>>>> Flink, but I couldn't get Spark to work.
>>>>
>>>> Here's a repo I've prepared for a bug reproduction, it may serve as
>>>> init:
>>>> https://github.com/orian/cogroup-wrong-grouping
>>>>
>>>> Could someone modify it or share some working example (outside of Beam
>>>> repo).
>>>>
>>>> Cheers, Pawel
>>>>
>>>
>>>
>>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Amit Sela <am...@gmail.com>.
Are you testing under package test ?
On Fri, Jun 3, 2016, 01:06 Pawel Szczur <pa...@gmail.com> wrote:
> I've just tested, it throws an exception with nightly Beam:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/JavaSparkContext
>
>
> 2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
>
>> You can find my pom.xml file for my beam-playground project that runs in
>> the spark runner, note that it uses the daily snapshots, but I will change
>> it for the released jars once they are on maven central:
>>
>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>
>> Regards,
>> Ismaël
>>
>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> A Beam can be easily configured to run against Direct, Dataflow and
>>> Flink, but I couldn't get Spark to work.
>>>
>>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>>> https://github.com/orian/cogroup-wrong-grouping
>>>
>>> Could someone modify it or share some working example (outside of Beam
>>> repo).
>>>
>>> Cheers, Pawel
>>>
>>
>>
>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Pawel Szczur <pa...@gmail.com>.
I've just tested, it throws an exception with nightly Beam:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/api/java/JavaSparkContext
2016-06-01 14:23 GMT+02:00 Ismaël Mejía <ie...@gmail.com>:
> You can find my pom.xml file for my beam-playground project that runs in
> the spark runner, note that it uses the daily snapshots, but I will change
> it for the released jars once they are on maven central:
>
> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
> Regards,
> Ismaël
>
> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> A Beam can be easily configured to run against Direct, Dataflow and
>> Flink, but I couldn't get Spark to work.
>>
>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>> https://github.com/orian/cogroup-wrong-grouping
>>
>> Could someone modify it or share some working example (outside of Beam
>> repo).
>>
>> Cheers, Pawel
>>
>
>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Pawel Szczur <pa...@gmail.com>.
Thanks Ismael, I was looking for something like that.
Amit: yes. It's only hard coded for the purpose of replicating a bug.
2016-06-01 14:38 GMT+02:00 Amit Sela <am...@gmail.com>:
> Did you try changing FlinkPipelineOptions and FlinkPipelineRunner to
> Spark's equivalents ? It looks like it's hard coded to run on Flink.
>
> On Wed, Jun 1, 2016 at 3:23 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You can find my pom.xml file for my beam-playground project that runs in
>> the spark runner, note that it uses the daily snapshots, but I will change
>> it for the released jars once they are on maven central:
>>
>> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>>
>> Regards,
>> Ismaël
>>
>> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> A Beam can be easily configured to run against Direct, Dataflow and
>>> Flink, but I couldn't get Spark to work.
>>>
>>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>>> https://github.com/orian/cogroup-wrong-grouping
>>>
>>> Could someone modify it or share some working example (outside of Beam
>>> repo).
>>>
>>> Cheers, Pawel
>>>
>>
>>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Amit Sela <am...@gmail.com>.
Did you try changing FlinkPipelineOptions and FlinkPipelineRunner to
Spark's equivalents ? It looks like it's hard coded to run on Flink.
On Wed, Jun 1, 2016 at 3:23 PM Ismaël Mejía <ie...@gmail.com> wrote:
> You can find my pom.xml file for my beam-playground project that runs in
> the spark runner, note that it uses the daily snapshots, but I will change
> it for the released jars once they are on maven central:
>
> https://github.com/iemejia/beam-playground/blob/master/pom.xml
>
> Regards,
> Ismaël
>
> On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> A Beam can be easily configured to run against Direct, Dataflow and
>> Flink, but I couldn't get Spark to work.
>>
>> Here's a repo I've prepared for a bug reproduction, it may serve as init:
>> https://github.com/orian/cogroup-wrong-grouping
>>
>> Could someone modify it or share some working example (outside of Beam
>> repo).
>>
>> Cheers, Pawel
>>
>
>
Re: Could someone share working bootstrap project with Spark runner configured?
Posted by Ismaël Mejía <ie...@gmail.com>.
You can find my pom.xml file for my beam-playground project that runs in
the spark runner, note that it uses the daily snapshots, but I will change
it for the released jars once they are on maven central:
https://github.com/iemejia/beam-playground/blob/master/pom.xml
Regards,
Ismaël
On Wed, Jun 1, 2016 at 3:01 AM, Pawel Szczur <pa...@gmail.com> wrote:
> Hi,
>
> A Beam can be easily configured to run against Direct, Dataflow and
> Flink, but I couldn't get Spark to work.
>
> Here's a repo I've prepared for a bug reproduction, it may serve as init:
> https://github.com/orian/cogroup-wrong-grouping
>
> Could someone modify it or share some working example (outside of Beam
> repo).
>
> Cheers, Pawel
>