You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Juan Carlos Garcia <jc...@gmail.com> on 2018/10/16 10:26:28 UTC
Error with FlinkRunner: No translator known for org.apache.beam.sdk.io.Read$Unbounded
Hi Folks,
I am switching some pipelines from SparkRunner to the FlinkRunner on beam
2.7
I started with a very simple pipeline which just reads from multiple kafka
sources, flatten those and then apply a regular DoFn.
On my first try of the pipeline (from command line using a fat jar) like:
```
java -cp my-fat.jar FQCN --configuration=foo --runner=FlinkRunner
```
complains with:
```
Exception in thread "main" java.lang.IllegalStateException: No translator
known for org.apache.beam.sdk.io.Read$Unbounded
at
org.apache.beam.runners.core.construction.PTransformTranslation.urnForTransform(PTransformTranslation.java:164)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:100)
...
...
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
```
On my build.gradle i just replaced my spark-runner with this:
``
runtime "org.apache.beam:beam-runners-flink_2.11:2.7.0"
``
Am i missing something?
--
JC
Re: Error with FlinkRunner: No translator known for
org.apache.beam.sdk.io.Read$Unbounded
Posted by Maximilian Michels <mx...@apache.org>.
Hi Juan,
Service files need to be merged when creating a fat jar because Beam
uses `ServiceLoader` at various places to register transforms. In your
case, only the special Flink transforms had been registered but not even
the default Beam transforms were picked up because their service files
were ignored.
Glad you were able to solve your problems.
-Max
On 16.10.18 13:27, Juan Carlos Garcia wrote:
> I figured it out, i was missing the 'mergeServiceFiles()' for the
> shadowJar gradle plugin, it looks like the SparkRunner was not needing
> such merge. (Or i was lucky, how the classes were arranged and processed
> when using the SparkRunner)
>
> Thanks.
>
>
>
> On Tue, Oct 16, 2018 at 1:09 PM Juan Carlos Garcia <jcgarciam@gmail.com
> <ma...@gmail.com>> wrote:
>
> I haven't yet resolved it but found out that my fat-jar (shadowJar)
> may have incomplete entries.
>
> Running the pipeline from within the IDE shows the following
> *KNOWN_PAYLOAD_TRANSLATOR*
> image.png
>
> But debugging the JVM process when launching the jar from the
> command line with
> (-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005)
>
> shows only the following *KNOWN_PAYLOAD_TRANSLATOR*
> image.png
>
> Will come back once i figure it out what configuration the shadowJar
> gradle plugin is missing.
>
>
> On Tue, Oct 16, 2018 at 12:26 PM Juan Carlos Garcia
> <jcgarciam@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Folks,
>
> I am switching some pipelines from SparkRunner to the
> FlinkRunner on beam 2.7
>
> I started with a very simple pipeline which just reads from
> multiple kafka sources, flatten those and then apply a regular DoFn.
>
> On my first try of the pipeline (from command line using a fat
> jar) like:
> ```
> java -cp my-fat.jar FQCN --configuration=foo --runner=FlinkRunner
> ```
>
> complains with:
> ```
> Exception in thread "main" java.lang.IllegalStateException: No
> translator known for org.apache.beam.sdk.io.Read$Unbounded
> at
> org.apache.beam.runners.core.construction.PTransformTranslation.urnForTransform(PTransformTranslation.java:164)
> at
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:100)
> ...
> ...
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> ```
>
> On my build.gradle i just replaced my spark-runner with this:
> ``
> runtime "org.apache.beam:beam-runners-flink_2.11:2.7.0"
> ``
>
> Am i missing something?
> --
>
> JC
>
>
>
> --
>
> JC
>
>
>
> --
>
> JC
>
Re: Error with FlinkRunner: No translator known for org.apache.beam.sdk.io.Read$Unbounded
Posted by Juan Carlos Garcia <jc...@gmail.com>.
I figured it out, i was missing the 'mergeServiceFiles()' for the shadowJar
gradle plugin, it looks like the SparkRunner was not needing such merge.
(Or i was lucky, how the classes were arranged and processed when using the
SparkRunner)
Thanks.
On Tue, Oct 16, 2018 at 1:09 PM Juan Carlos Garcia <jc...@gmail.com>
wrote:
> I haven't yet resolved it but found out that my fat-jar (shadowJar) may
> have incomplete entries.
>
> Running the pipeline from within the IDE shows the following
> *KNOWN_PAYLOAD_TRANSLATOR*
> [image: image.png]
>
> But debugging the JVM process when launching the jar from the command line
> with (-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005)
>
> shows only the following *KNOWN_PAYLOAD_TRANSLATOR*
> [image: image.png]
>
> Will come back once i figure it out what configuration the shadowJar
> gradle plugin is missing.
>
>
> On Tue, Oct 16, 2018 at 12:26 PM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> Hi Folks,
>>
>> I am switching some pipelines from SparkRunner to the FlinkRunner on beam
>> 2.7
>>
>> I started with a very simple pipeline which just reads from multiple
>> kafka sources, flatten those and then apply a regular DoFn.
>>
>> On my first try of the pipeline (from command line using a fat jar) like:
>> ```
>> java -cp my-fat.jar FQCN --configuration=foo --runner=FlinkRunner
>> ```
>>
>> complains with:
>> ```
>> Exception in thread "main" java.lang.IllegalStateException: No translator
>> known for org.apache.beam.sdk.io.Read$Unbounded
>> at
>> org.apache.beam.runners.core.construction.PTransformTranslation.urnForTransform(PTransformTranslation.java:164)
>> at
>> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:100)
>> ...
>> ...
>> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
>> ```
>>
>> On my build.gradle i just replaced my spark-runner with this:
>> ``
>> runtime "org.apache.beam:beam-runners-flink_2.11:2.7.0"
>> ``
>>
>> Am i missing something?
>> --
>>
>> JC
>>
>>
>
> --
>
> JC
>
>
--
JC
Re: Error with FlinkRunner: No translator known for org.apache.beam.sdk.io.Read$Unbounded
Posted by Juan Carlos Garcia <jc...@gmail.com>.
I haven't yet resolved it but found out that my fat-jar (shadowJar) may
have incomplete entries.
Running the pipeline from within the IDE shows the following
*KNOWN_PAYLOAD_TRANSLATOR*
[image: image.png]
But debugging the JVM process when launching the jar from the command line
with (-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005)
shows only the following *KNOWN_PAYLOAD_TRANSLATOR*
[image: image.png]
Will come back once i figure it out what configuration the shadowJar gradle
plugin is missing.
On Tue, Oct 16, 2018 at 12:26 PM Juan Carlos Garcia <jc...@gmail.com>
wrote:
> Hi Folks,
>
> I am switching some pipelines from SparkRunner to the FlinkRunner on beam
> 2.7
>
> I started with a very simple pipeline which just reads from multiple kafka
> sources, flatten those and then apply a regular DoFn.
>
> On my first try of the pipeline (from command line using a fat jar) like:
> ```
> java -cp my-fat.jar FQCN --configuration=foo --runner=FlinkRunner
> ```
>
> complains with:
> ```
> Exception in thread "main" java.lang.IllegalStateException: No translator
> known for org.apache.beam.sdk.io.Read$Unbounded
> at
> org.apache.beam.runners.core.construction.PTransformTranslation.urnForTransform(PTransformTranslation.java:164)
> at
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:100)
> ...
> ...
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> ```
>
> On my build.gradle i just replaced my spark-runner with this:
> ``
> runtime "org.apache.beam:beam-runners-flink_2.11:2.7.0"
> ``
>
> Am i missing something?
> --
>
> JC
>
>
--
JC