You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by amir bahmanyari <am...@yahoo.com> on 2016/05/09 22:11:06 UTC

KafkaIO call WithoutMetadata is currently not supported?

Hi Colleagues,I get this exception at runtime...Any ideas pls?Thanks

java.lang.UnsupportedOperationException: The transform KafkaIO.Read [KafkaIO.TypedWithoutMetadata] is currently not supported.        at org.apache.beam.runners.flink.translation.FlinkStreamingPipelineTranslator.visitTransform(FlinkStreamingPipelineTranslator.java:106)

Re: KafkaIO call WithoutMetadata is currently not supported?

Posted by Maximilian Michels <mx...@apache.org>.
Hi Amir,

BEAM-103 affects the runtime execution of the Beam sources (now
parallelized). The error above occurs during the translation phase of the
Beam program into the Flink API. I couldn't reproduce the error. I ran the
following program and it printed incoming Kafka records:

FlinkPipelineOptions options =
PipelineOptionsFactory.fromArgs(args).as(FlinkPipelineOptions.class);
options.setStreaming(true);
options.setRunner(FlinkPipelineRunner.class);

final Pipeline p = Pipeline.create(options);

final PCollection<KV<String, byte[]>> input = p.apply(KafkaIO.read()
    .withTopics(ImmutableList.of("test"))
    .withKeyCoder(StringUtf8Coder.of())
    .withBootstrapServers("localhost:9092")
    .withoutMetadata());

input.apply(ParDo.of(new DoFn<KV<String, byte[]>, String>() {
  @Override
  public void processElement(ProcessContext c) throws Exception {
    System.out.println("element: " + c.element());
  }
}));

p.run();


Note, that this is Kafka 9.

Cheers,
Max



On Mon, May 16, 2016 at 6:48 AM, Frances Perry <fj...@google.com> wrote:

> This might have been https://issues.apache.org/jira/browse/BEAM-103 ?
>
> But if so, the good news it was fixed four days ago ;-)
>
> On Mon, May 9, 2016 at 3:11 PM, amir bahmanyari <am...@yahoo.com>
> wrote:
>
>> Hi Colleagues,
>> I get this exception at runtime...
>> Any ideas pls?
>> Thanks
>>
>>
>> java.lang.UnsupportedOperationException: The transform KafkaIO.Read
>> [KafkaIO.TypedWithoutMetadata] is currently not supported.
>>         at
>> org.apache.beam.runners.flink.translation.FlinkStreamingPipelineTranslator.visitTransform(FlinkStreamingPipelineTranslator.java:106)
>>
>>
>

Re: KafkaIO call WithoutMetadata is currently not supported?

Posted by Frances Perry <fj...@google.com>.
This might have been https://issues.apache.org/jira/browse/BEAM-103 ?

But if so, the good news it was fixed four days ago ;-)

On Mon, May 9, 2016 at 3:11 PM, amir bahmanyari <am...@yahoo.com> wrote:

> Hi Colleagues,
> I get this exception at runtime...
> Any ideas pls?
> Thanks
>
>
> java.lang.UnsupportedOperationException: The transform KafkaIO.Read
> [KafkaIO.TypedWithoutMetadata] is currently not supported.
>         at
> org.apache.beam.runners.flink.translation.FlinkStreamingPipelineTranslator.visitTransform(FlinkStreamingPipelineTranslator.java:106)
>
>