You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ismaël Mejía <ie...@gmail.com> on 2016/05/18 10:28:36 UTC
issue with reusing spark context in spark runner
Hello,
I am trying to run a set of tests that use the spark runner, I build the
Pipeline in a setUp method and then reuse it in different tests, however
when it is invoked for the second time it throws an exception:
java.lang.RuntimeException: org.apache.spark.SparkException: Only one
SparkContext may be running in this JVM (see SPARK-2243). To ignore this
error, set spark.driver.allowMultipleContexts = true. The currently running
SparkContext was created at:
Do you know how can I pass such variable to the runner, or if I can skip
this issue in another way ?
-Ismael
Re: issue with reusing spark context in spark runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks Amit, it worked perfect !
On Wed, May 18, 2016 at 12:33 PM, Amit Sela <am...@gmail.com> wrote:
> You can pass this system property: "
> dataflow.spark.test.reuseSparkContext=true"
> And this will reuse the context, see:
> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
>
> On Wed, May 18, 2016 at 1:28 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Hello,
>>
>> I am trying to run a set of tests that use the spark runner, I build the
>> Pipeline in a setUp method and then reuse it in different tests, however
>> when it is invoked for the second time it throws an exception:
>>
>> java.lang.RuntimeException: org.apache.spark.SparkException: Only one
>> SparkContext may be running in this JVM (see SPARK-2243). To ignore this
>> error, set spark.driver.allowMultipleContexts = true. The currently running
>> SparkContext was created at:
>>
>> Do you know how can I pass such variable to the runner, or if I can skip
>> this issue in another way ?
>>
>> -Ismael
>>
>
Re: issue with reusing spark context in spark runner
Posted by Amit Sela <am...@gmail.com>.
Yep. Sorry, I stepped in while I was out of context 😉
On Wed, May 18, 2016, 16:51 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> As I said, I already did the change on a branch.
>
> Let me submit the PR.
>
> Regards
> JB
>
> On 05/18/2016 03:40 PM, Ismaël Mejía wrote:
> > I can take care of that, and other 'dataflow' mentions in the runner if
> > you agree.
> >
> > On Wed, May 18, 2016 at 2:30 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > We should change the property name by the way (dataflow => beam).
> >
> > I think I already did it on a local branch.
> >
> > Regards
> > JB
> >
> > On 05/18/2016 12:33 PM, Amit Sela wrote:
> >
> > You can pass this system property:
> > "dataflow.spark.test.reuseSparkContext=true"
> > And this will reuse the context, see:
> >
> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
> >
> > On Wed, May 18, 2016 at 1:28 PM Ismaël Mejía <iemejia@gmail.com
> > <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Hello,
> >
> > I am trying to run a set of tests that use the spark
> > runner, I build
> > the Pipeline in a setUp method and then reuse it in
> > different tests,
> > however when it is invoked for the second time it throws an
> > exception:
> >
> > java.lang.RuntimeException:
> > org.apache.spark.SparkException: Only
> > one SparkContext may be running in this JVM (see
> > SPARK-2243). To
> > ignore this error, set spark.driver.allowMultipleContexts =
> > true.
> > The currently running SparkContext was created at:
> >
> > Do you know how can I pass such variable to the runner, or
> > if I can
> > skip this issue in another way ?
> >
> > -Ismael
> >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: issue with reusing spark context in spark runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
As I said, I already did the change on a branch.
Let me submit the PR.
Regards
JB
On 05/18/2016 03:40 PM, Isma�l Mej�a wrote:
> I can take care of that, and other 'dataflow' mentions in the runner if
> you agree.
>
> On Wed, May 18, 2016 at 2:30 PM, Jean-Baptiste Onofr� <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> We should change the property name by the way (dataflow => beam).
>
> I think I already did it on a local branch.
>
> Regards
> JB
>
> On 05/18/2016 12:33 PM, Amit Sela wrote:
>
> You can pass this system property:
> "dataflow.spark.test.reuseSparkContext=true"
> And this will reuse the context, see:
> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
>
> On Wed, May 18, 2016 at 1:28 PM Isma�l Mej�a <iemejia@gmail.com
> <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>> wrote:
>
> Hello,
>
> I am trying to run a set of tests that use the spark
> runner, I build
> the Pipeline in a setUp method and then reuse it in
> different tests,
> however when it is invoked for the second time it throws an
> exception:
>
> java.lang.RuntimeException:
> org.apache.spark.SparkException: Only
> one SparkContext may be running in this JVM (see
> SPARK-2243). To
> ignore this error, set spark.driver.allowMultipleContexts =
> true.
> The currently running SparkContext was created at:
>
> Do you know how can I pass such variable to the runner, or
> if I can
> skip this issue in another way ?
>
> -Ismael
>
>
> --
> Jean-Baptiste Onofr�
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: issue with reusing spark context in spark runner
Posted by Amit Sela <am...@gmail.com>.
Sure!
On Wed, May 18, 2016, 16:40 Ismaël Mejía <ie...@gmail.com> wrote:
> I can take care of that, and other 'dataflow' mentions in the runner if
> you agree.
>
> On Wed, May 18, 2016 at 2:30 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> We should change the property name by the way (dataflow => beam).
>>
>> I think I already did it on a local branch.
>>
>> Regards
>> JB
>>
>> On 05/18/2016 12:33 PM, Amit Sela wrote:
>>
>>> You can pass this system property:
>>> "dataflow.spark.test.reuseSparkContext=true"
>>> And this will reuse the context, see:
>>>
>>> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
>>>
>>> On Wed, May 18, 2016 at 1:28 PM Ismaël Mejía <iemejia@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run a set of tests that use the spark runner, I build
>>> the Pipeline in a setUp method and then reuse it in different tests,
>>> however when it is invoked for the second time it throws an
>>> exception:
>>>
>>> java.lang.RuntimeException: org.apache.spark.SparkException: Only
>>> one SparkContext may be running in this JVM (see SPARK-2243). To
>>> ignore this error, set spark.driver.allowMultipleContexts = true.
>>> The currently running SparkContext was created at:
>>>
>>> Do you know how can I pass such variable to the runner, or if I can
>>> skip this issue in another way ?
>>>
>>> -Ismael
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
Re: issue with reusing spark context in spark runner
Posted by Ismaël Mejía <ie...@gmail.com>.
I can take care of that, and other 'dataflow' mentions in the runner if you
agree.
On Wed, May 18, 2016 at 2:30 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> We should change the property name by the way (dataflow => beam).
>
> I think I already did it on a local branch.
>
> Regards
> JB
>
> On 05/18/2016 12:33 PM, Amit Sela wrote:
>
>> You can pass this system property:
>> "dataflow.spark.test.reuseSparkContext=true"
>> And this will reuse the context, see:
>>
>> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
>>
>> On Wed, May 18, 2016 at 1:28 PM Ismaël Mejía <iemejia@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hello,
>>
>> I am trying to run a set of tests that use the spark runner, I build
>> the Pipeline in a setUp method and then reuse it in different tests,
>> however when it is invoked for the second time it throws an exception:
>>
>> java.lang.RuntimeException: org.apache.spark.SparkException: Only
>> one SparkContext may be running in this JVM (see SPARK-2243). To
>> ignore this error, set spark.driver.allowMultipleContexts = true.
>> The currently running SparkContext was created at:
>>
>> Do you know how can I pass such variable to the runner, or if I can
>> skip this issue in another way ?
>>
>> -Ismael
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: issue with reusing spark context in spark runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
We should change the property name by the way (dataflow => beam).
I think I already did it on a local branch.
Regards
JB
On 05/18/2016 12:33 PM, Amit Sela wrote:
> You can pass this system property:
> "dataflow.spark.test.reuseSparkContext=true"
> And this will reuse the context, see:
> https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
>
> On Wed, May 18, 2016 at 1:28 PM Isma�l Mej�a <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello,
>
> I am trying to run a set of tests that use the spark runner, I build
> the Pipeline in a setUp method and then reuse it in different tests,
> however when it is invoked for the second time it throws an exception:
>
> java.lang.RuntimeException: org.apache.spark.SparkException: Only
> one SparkContext may be running in this JVM (see SPARK-2243). To
> ignore this error, set spark.driver.allowMultipleContexts = true.
> The currently running SparkContext was created at:
>
> Do you know how can I pass such variable to the runner, or if I can
> skip this issue in another way ?
>
> -Ismael
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: issue with reusing spark context in spark runner
Posted by Amit Sela <am...@gmail.com>.
You can pass this system property: "
dataflow.spark.test.reuseSparkContext=true"
And this will reuse the context, see:
https://github.com/apache/incubator-beam/blob/d627266d8d39ff0ec94dc9f3f84893c1026abde7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L35
On Wed, May 18, 2016 at 1:28 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Hello,
>
> I am trying to run a set of tests that use the spark runner, I build the
> Pipeline in a setUp method and then reuse it in different tests, however
> when it is invoked for the second time it throws an exception:
>
> java.lang.RuntimeException: org.apache.spark.SparkException: Only one
> SparkContext may be running in this JVM (see SPARK-2243). To ignore this
> error, set spark.driver.allowMultipleContexts = true. The currently running
> SparkContext was created at:
>
> Do you know how can I pass such variable to the runner, or if I can skip
> this issue in another way ?
>
> -Ismael
>