You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Juan Carlos Garcia <jc...@gmail.com> on 2018/10/04 05:18:31 UTC
Re: Beam SparkRunner and Spark KryoSerializer problem
Bump,
can someone from the core-dev provide a feedback about:
https://issues.apache.org/jira/browse/BEAM-4597
Thanks
On Mon, Jul 30, 2018 at 3:15 PM Juan Carlos Garcia <jc...@gmail.com>
wrote:
> Hi Jean,
>
> Thanks for taking a look.
>
>
> On Mon, Jul 30, 2018 at 2:49 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Juan,
>>
>> it seems that has been introduce by the metrics layer in the core runner
>> API.
>>
>> Let me check.
>>
>> Regards
>> JB
>>
>> On 30/07/2018 14:47, Juan Carlos Garcia wrote:
>> > Bump!
>> >
>> > Does any of the core-dev roam around here?
>> >
>> > Can someone provide a feedback about BEAM-4597
>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>> >
>> > Thanks and regards,
>> >
>> > On Thu, Jul 19, 2018 at 3:41 PM, Juan Carlos Garcia <
>> jcgarciam@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Folks,
>> >
>> > Its someone using the SparkRunner out there with the Spark
>> > KryoSerializer ?
>> >
>> > We are being force to use the not so efficient 'JavaSerializer' with
>> > Spark because we face the following exception:
>> >
>> > <exception>
>> > Exception in thread "main" java.lang.RuntimeException:
>> > org.apache.spark.SparkException: Job aborted due to stage failure:
>> > Exception while getting task result:
>> > com.esotericsoftware.kryo.KryoException: Unable to find class:
>> >
>> org.apache.beam.runners.core.metrics.MetricsContainerImpl$$Lambda$31/1875283985
>> > Serialization trace:
>> > factory (org.apache.beam.runners.core.metrics.MetricsMap)
>> > counters (org.apache.beam.runners.core.metrics.MetricsContainerImpl)
>> > metricsContainers
>> > (org.apache.beam.runners.core.metrics.MetricsContainerStepMap)
>> > metricsContainers
>> > (org.apache.beam.runners.spark.io.SparkUnboundedSource$Metadata)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:55)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.access$000(SparkPipelineResult.java:41)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult$StreamingMode.stop(SparkPipelineResult.java:163)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.offerNewState(SparkPipelineResult.java:198)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:101)
>> > at
>> >
>> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87)
>> > at
>> >
>> org.apache.beam.examples.BugWithKryoOnSpark.main(BugWithKryoOnSpark.java:75)
>> > </exception>
>> >
>> > I created a jira ticket and attached a project example on it,
>> > https://issues.apache.org/jira/browse/BEAM-4597
>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>> >
>> > Any feedback is appreciated.
>> >
>> > --
>> >
>> > JC
>> >
>> >
>> >
>> >
>> > --
>> >
>> > JC
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
>
> --
>
> JC
>
>
--
JC
Re: Beam SparkRunner and Spark KryoSerializer problem
Posted by Juan Carlos Garcia <jc...@gmail.com>.
Hi Jean,
Thank you!
On Thu, Oct 4, 2018 at 7:54 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Juan
>
> I'm on it.
>
> Regards
> JB
> Le 4 oct. 2018, à 07:19, Juan Carlos Garcia <jc...@gmail.com> a écrit:
>>
>> Bump,
>>
>> can someone from the core-dev provide a feedback about:
>> https://issues.apache.org/jira/browse/BEAM-4597
>>
>> Thanks
>>
>> On Mon, Jul 30, 2018 at 3:15 PM Juan Carlos Garcia <jc...@gmail.com>
>> wrote:
>>
>>> Hi Jean,
>>>
>>> Thanks for taking a look.
>>>
>>>
>>> On Mon, Jul 30, 2018 at 2:49 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi Juan,
>>>>
>>>> it seems that has been introduce by the metrics layer in the core runner
>>>> API.
>>>>
>>>> Let me check.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 30/07/2018 14:47, Juan Carlos Garcia wrote:
>>>> > Bump!
>>>> >
>>>> > Does any of the core-dev roam around here?
>>>> >
>>>> > Can someone provide a feedback about BEAM-4597
>>>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>>>> >
>>>> > Thanks and regards,
>>>> >
>>>> > On Thu, Jul 19, 2018 at 3:41 PM, Juan Carlos Garcia <
>>>> jcgarciam@gmail.com
>>>> > <ma...@gmail.com>> wrote:
>>>> >
>>>> > Folks,
>>>> >
>>>> > Its someone using the SparkRunner out there with the Spark
>>>> > KryoSerializer ?
>>>> >
>>>> > We are being force to use the not so efficient 'JavaSerializer'
>>>> with
>>>> > Spark because we face the following exception:
>>>> >
>>>> > <exception>
>>>> > Exception in thread "main" java.lang.RuntimeException:
>>>> > org.apache.spark.SparkException: Job aborted due to stage failure:
>>>> > Exception while getting task result:
>>>> > com.esotericsoftware.kryo.KryoException: Unable to find class:
>>>> >
>>>> org.apache.beam.runners.core.metrics.MetricsContainerImpl$$Lambda$31/1875283985
>>>> > Serialization trace:
>>>> > factory (org.apache.beam.runners.core.metrics.MetricsMap)
>>>> > counters
>>>> (org.apache.beam.runners.core.metrics.MetricsContainerImpl)
>>>> > metricsContainers
>>>> > (org.apache.beam.runners.core.metrics.MetricsContainerStepMap)
>>>> > metricsContainers
>>>> > (org.apache.beam.runners.spark.io.SparkUnboundedSource$Metadata)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:55)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.access$000(SparkPipelineResult.java:41)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult$StreamingMode.stop(SparkPipelineResult.java:163)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.offerNewState(SparkPipelineResult.java:198)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:101)
>>>> > at
>>>> >
>>>> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87)
>>>> > at
>>>> >
>>>> org.apache.beam.examples.BugWithKryoOnSpark.main(BugWithKryoOnSpark.java:75)
>>>> > </exception>
>>>> >
>>>> > I created a jira ticket and attached a project example on it,
>>>> > https://issues.apache.org/jira/browse/BEAM-4597
>>>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>>>> >
>>>> > Any feedback is appreciated.
>>>> >
>>>> > --
>>>> >
>>>> > JC
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > JC
>>>> >
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> JC
>>>
>>>
>>
>> --
>>
>> JC
>>
>>
--
JC
Re: Beam SparkRunner and Spark KryoSerializer problem
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Juan
I'm on it.
Regards
JB
Le 4 oct. 2018 à 07:19, à 07:19, Juan Carlos Garcia <jc...@gmail.com> a écrit:
>Bump,
>
>can someone from the core-dev provide a feedback about:
> https://issues.apache.org/jira/browse/BEAM-4597
>
>Thanks
>
>On Mon, Jul 30, 2018 at 3:15 PM Juan Carlos Garcia
><jc...@gmail.com>
>wrote:
>
>> Hi Jean,
>>
>> Thanks for taking a look.
>>
>>
>> On Mon, Jul 30, 2018 at 2:49 PM, Jean-Baptiste Onofré
><jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Juan,
>>>
>>> it seems that has been introduce by the metrics layer in the core
>runner
>>> API.
>>>
>>> Let me check.
>>>
>>> Regards
>>> JB
>>>
>>> On 30/07/2018 14:47, Juan Carlos Garcia wrote:
>>> > Bump!
>>> >
>>> > Does any of the core-dev roam around here?
>>> >
>>> > Can someone provide a feedback about BEAM-4597
>>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>>> >
>>> > Thanks and regards,
>>> >
>>> > On Thu, Jul 19, 2018 at 3:41 PM, Juan Carlos Garcia <
>>> jcgarciam@gmail.com
>>> > <ma...@gmail.com>> wrote:
>>> >
>>> > Folks,
>>> >
>>> > Its someone using the SparkRunner out there with the Spark
>>> > KryoSerializer ?
>>> >
>>> > We are being force to use the not so efficient
>'JavaSerializer' with
>>> > Spark because we face the following exception:
>>> >
>>> > <exception>
>>> > Exception in thread "main" java.lang.RuntimeException:
>>> > org.apache.spark.SparkException: Job aborted due to stage
>failure:
>>> > Exception while getting task result:
>>> > com.esotericsoftware.kryo.KryoException: Unable to find class:
>>> >
>>>
>org.apache.beam.runners.core.metrics.MetricsContainerImpl$$Lambda$31/1875283985
>>> > Serialization trace:
>>> > factory (org.apache.beam.runners.core.metrics.MetricsMap)
>>> > counters
>(org.apache.beam.runners.core.metrics.MetricsContainerImpl)
>>> > metricsContainers
>>> > (org.apache.beam.runners.core.metrics.MetricsContainerStepMap)
>>> > metricsContainers
>>> >
>(org.apache.beam.runners.spark.io.SparkUnboundedSource$Metadata)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:55)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.access$000(SparkPipelineResult.java:41)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult$StreamingMode.stop(SparkPipelineResult.java:163)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.offerNewState(SparkPipelineResult.java:198)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:101)
>>> > at
>>> >
>>>
>org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87)
>>> > at
>>> >
>>>
>org.apache.beam.examples.BugWithKryoOnSpark.main(BugWithKryoOnSpark.java:75)
>>> > </exception>
>>> >
>>> > I created a jira ticket and attached a project example on it,
>>> > https://issues.apache.org/jira/browse/BEAM-4597
>>> > <https://issues.apache.org/jira/browse/BEAM-4597>
>>> >
>>> > Any feedback is appreciated.
>>> >
>>> > --
>>> >
>>> > JC
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > JC
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>>
>>
>> --
>>
>> JC
>>
>>
>
>--
>
>JC