You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashutosh Kumar <km...@gmail.com> on 2016/11/02 11:48:09 UTC

Running Google Dataflow on Spark

I am trying to run Google Dataflow code on Spark. It works fine as google
dataflow on google cloud platform. But while running on Spark I am getting
following error

16/11/02 11:14:32 INFO com.cloudera.dataflow.spark.SparkPipelineRunner:
Evaluating ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Exception in thread "main" java.lang.RuntimeException:
java.lang.IllegalStateException: No TransformEvaluator registered for class
com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner$Grou
pByKeyAndSortValuesOnly
        at
com.cloudera.dataflow.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:124)

Thanks
Ashutosh

Re: Running Google Dataflow on Spark

Posted by Sean Owen <so...@cloudera.com>.
This is a Dataflow / Beam question, not a Spark question per se.

On Wed, Nov 2, 2016 at 11:48 AM Ashutosh Kumar <km...@gmail.com>
wrote:

> I am trying to run Google Dataflow code on Spark. It works fine as google
> dataflow on google cloud platform. But while running on Spark I am getting
> following error
>
> 16/11/02 11:14:32 INFO com.cloudera.dataflow.spark.SparkPipelineRunner:
> Evaluating ParDo(GroupByKeyHashAndSortByKeyAndWindow)
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.IllegalStateException: No TransformEvaluator registered for class
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner$Grou
> pByKeyAndSortValuesOnly
>         at
> com.cloudera.dataflow.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:124)
>
> Thanks
> Ashutosh
>