You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Jesse Anderson <je...@smokinghand.com> on 2016/09/20 00:39:30 UTC
Spark TransformEvaluator Error
I'm running this Beam program
<https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>.
When I run it on Spark, it get this error:
Exception in thread "main" java.lang.RuntimeException:
java.lang.IllegalStateException: No TransformEvaluator registered for class
org.apache.beam.sdk.io.Read$Bounded
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
at
org.apache.beam.examples.tutorial.game.solution.Exercise2.main(Exercise2.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: No TransformEvaluator
registered for class org.apache.beam.sdk.io.Read$Bounded
at
org.apache.beam.runners.spark.translation.TransformTranslator.getTransformEvaluator(TransformTranslator.java:892)
at
org.apache.beam.runners.spark.translation.TransformTranslator$Translator.translate(TransformTranslator.java:910)
at
org.apache.beam.runners.spark.translation.SparkPipelineEvaluator.doVisitTransform(SparkPipelineEvaluator.java:49)
at
org.apache.beam.runners.spark.SparkRunner$Evaluator.visitPrimitiveTransform(SparkRunner.java:242)
at
org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:224)
at
org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
at
org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:101)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179)
... 12 more
Looking at the TransformTranslator class, I don't see that
org.apache.beam.sdk.io.Read
is added. Is there a call I need to make to add Read?
Thanks,
Jesse
Re: Spark TransformEvaluator Error
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
There's a work in progress (on a branch) to support bounded/unbounded read and write.
Amit did it and I'm testing/reviewing it (so far so good).
Regards
JB
On Sep 19, 2016, 20:52, at 20:52, Lukasz Cwik <lc...@google.com> wrote:
>The Spark runner currently only supports a limited set of sources
>because
>there is no support in the Spark runner implementation which handles
>the
>Read PTransform directly.
>
>They are able to support TextIO, AvroIO and a few others by manually
>providing a few direct implementations.
>
>The supported list is here:
>https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java#L865
>
>
>
>On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <je...@smokinghand.com>
>wrote:
>
>> I'm running this Beam program
>>
><https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>.
>> When I run it on Spark, it get this error:
>>
>> Exception in thread "main" java.lang.RuntimeException:
>java.lang.IllegalStateException:
>> No TransformEvaluator registered for class
>org.apache.beam.sdk.io.Read$
>> Bounded
>> at
>org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201)
>> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71)
>> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>> at org.apache.beam.examples.tutorial.game.solution.
>> Exercise2.main(Exercise2.java:91)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
>> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>> at
>org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.IllegalStateException: No TransformEvaluator
>> registered for class org.apache.beam.sdk.io.Read$Bounded
>> at org.apache.beam.runners.spark.translation.TransformTranslator.
>> getTransformEvaluator(TransformTranslator.java:892)
>> at org.apache.beam.runners.spark.translation.TransformTranslator$
>> Translator.translate(TransformTranslator.java:910)
>> at org.apache.beam.runners.spark.translation.SparkPipelineEvaluator.
>> doVisitTransform(SparkPipelineEvaluator.java:49)
>> at org.apache.beam.runners.spark.SparkRunner$Evaluator.
>> visitPrimitiveTransform(SparkRunner.java:242)
>> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
>> TransformTreeNode.java:224)
>> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
>> TransformTreeNode.java:219)
>> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
>> TransformTreeNode.java:219)
>> at org.apache.beam.sdk.runners.TransformHierarchy.visit(
>> TransformHierarchy.java:101)
>> at
>org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
>> at
>org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179)
>> ... 12 more
>>
>> Looking at the TransformTranslator class, I don't see that
>> org.apache.beam.sdk.io.Read is added. Is there a call I need to make
>to
>> add Read?
>>
>> Thanks,
>>
>> Jesse
>>
>>
Re: Spark TransformEvaluator Error
Posted by Lukasz Cwik <lc...@google.com>.
The Spark runner currently only supports a limited set of sources because
there is no support in the Spark runner implementation which handles the
Read PTransform directly.
They are able to support TextIO, AvroIO and a few others by manually
providing a few direct implementations.
The supported list is here:
https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java#L865
On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <je...@smokinghand.com>
wrote:
> I'm running this Beam program
> <https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>.
> When I run it on Spark, it get this error:
>
> Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalStateException:
> No TransformEvaluator registered for class org.apache.beam.sdk.io.Read$
> Bounded
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
> at org.apache.beam.examples.tutorial.game.solution.
> Exercise2.main(Exercise2.java:91)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.IllegalStateException: No TransformEvaluator
> registered for class org.apache.beam.sdk.io.Read$Bounded
> at org.apache.beam.runners.spark.translation.TransformTranslator.
> getTransformEvaluator(TransformTranslator.java:892)
> at org.apache.beam.runners.spark.translation.TransformTranslator$
> Translator.translate(TransformTranslator.java:910)
> at org.apache.beam.runners.spark.translation.SparkPipelineEvaluator.
> doVisitTransform(SparkPipelineEvaluator.java:49)
> at org.apache.beam.runners.spark.SparkRunner$Evaluator.
> visitPrimitiveTransform(SparkRunner.java:242)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:224)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:219)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:219)
> at org.apache.beam.sdk.runners.TransformHierarchy.visit(
> TransformHierarchy.java:101)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179)
> ... 12 more
>
> Looking at the TransformTranslator class, I don't see that
> org.apache.beam.sdk.io.Read is added. Is there a call I need to make to
> add Read?
>
> Thanks,
>
> Jesse
>
>
Re: Spark TransformEvaluator Error
Posted by Amit Sela <am...@gmail.com>.
This is a WIP. I'm aware of Strata and will do my best to have this ready
ASAP.
Thanks,
Amit.
On Tue, Sep 20, 2016, 06:53 Frances Perry <fj...@google.com> wrote:
> +amitsela@apache.org
>
> From https://issues.apache.org/jira/browse/BEAM-17, it looks like the
> Spark runner doesn't support that yet.
>
> On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <je...@smokinghand.com>
> wrote:
>
>> I'm running this Beam program
>> <https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>.
>> When I run it on Spark, it get this error:
>>
>> Exception in thread "main" java.lang.RuntimeException:
>> java.lang.IllegalStateException: No TransformEvaluator registered for class
>> org.apache.beam.sdk.io.Read$Bounded
>> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201)
>> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71)
>> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>> at
>> org.apache.beam.examples.tutorial.game.solution.Exercise2.main(Exercise2.java:91)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.IllegalStateException: No TransformEvaluator
>> registered for class org.apache.beam.sdk.io.Read$Bounded
>> at
>> org.apache.beam.runners.spark.translation.TransformTranslator.getTransformEvaluator(TransformTranslator.java:892)
>> at
>> org.apache.beam.runners.spark.translation.TransformTranslator$Translator.translate(TransformTranslator.java:910)
>> at
>> org.apache.beam.runners.spark.translation.SparkPipelineEvaluator.doVisitTransform(SparkPipelineEvaluator.java:49)
>> at
>> org.apache.beam.runners.spark.SparkRunner$Evaluator.visitPrimitiveTransform(SparkRunner.java:242)
>> at
>> org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:224)
>> at
>> org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
>> at
>> org.apache.beam.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
>> at
>> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:101)
>> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
>> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179)
>> ... 12 more
>>
>> Looking at the TransformTranslator class, I don't see that
>> org.apache.beam.sdk.io.Read is added. Is there a call I need to make to
>> add Read?
>>
>> Thanks,
>>
>> Jesse
>>
>>
Re: Spark TransformEvaluator Error
Posted by Frances Perry <fj...@google.com>.
+amitsela@apache.org
From https://issues.apache.org/jira/browse/BEAM-17, it looks like the Spark
runner doesn't support that yet.
On Mon, Sep 19, 2016 at 5:39 PM, Jesse Anderson <je...@smokinghand.com>
wrote:
> I'm running this Beam program
> <https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game/utils/Input.java#L38>.
> When I run it on Spark, it get this error:
>
> Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalStateException:
> No TransformEvaluator registered for class org.apache.beam.sdk.io.Read$
> Bounded
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:201)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:71)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
> at org.apache.beam.examples.tutorial.game.solution.
> Exercise2.main(Exercise2.java:91)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.IllegalStateException: No TransformEvaluator
> registered for class org.apache.beam.sdk.io.Read$Bounded
> at org.apache.beam.runners.spark.translation.TransformTranslator.
> getTransformEvaluator(TransformTranslator.java:892)
> at org.apache.beam.runners.spark.translation.TransformTranslator$
> Translator.translate(TransformTranslator.java:910)
> at org.apache.beam.runners.spark.translation.SparkPipelineEvaluator.
> doVisitTransform(SparkPipelineEvaluator.java:49)
> at org.apache.beam.runners.spark.SparkRunner$Evaluator.
> visitPrimitiveTransform(SparkRunner.java:242)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:224)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:219)
> at org.apache.beam.sdk.runners.TransformTreeNode.visit(
> TransformTreeNode.java:219)
> at org.apache.beam.sdk.runners.TransformHierarchy.visit(
> TransformHierarchy.java:101)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:292)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:179)
> ... 12 more
>
> Looking at the TransformTranslator class, I don't see that
> org.apache.beam.sdk.io.Read is added. Is there a call I need to make to
> add Read?
>
> Thanks,
>
> Jesse
>
>