You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Boyuan Zhang (Jira)" <ji...@apache.org> on 2020/08/11 17:01:00 UTC
[jira] [Comment Edited] (BEAM-6868) Flink runner supports Bundle
Finalization
[ https://issues.apache.org/jira/browse/BEAM-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175705#comment-17175705 ]
Boyuan Zhang edited comment on BEAM-6868 at 8/11/20, 5:00 PM:
--------------------------------------------------------------
Thanks. Cham! There are 2 different errors mentioned on the https://lists.apache.org/thread.html/re53d28276def9593dc5f28cbd9710d141aa0b68e2e0486d401724398%40%3Cdev.beam.apache.org%3E thread:
* The ActiveBundle does not have a registered *bundle finalization handler*. This means the SDK DoFn requests bundle finalization but the runner doesn't support. In x-lang kafka read, it's required by the unbounded sdf wrapper.
* The ActiveBundle does not have a registered bundle *checkpoint handler* This means the SDK DoFn returns a ProcessContinuation and may perform self-checkpoint during execution time but the runner doesn't handle DelayedBundleApplication. In x-lang kafka read, it's required by the sdf Kafka implementation.
In order to use x-lang Kafka with flink runner, we can:
* either support bundle finalization in flink meanwhile use sdf wrapper for Kafka.
* Or we can support self-checkpoint in flink and use sdf Kafka instead.
But in long term, it would be nice to have both in flink runner.
was (Author: boyuanz):
Thanks. Cham! There are 2 different errors mentioned on the https://lists.apache.org/thread.html/re53d28276def9593dc5f28cbd9710d141aa0b68e2e0486d401724398%40%3Cdev.beam.apache.org%3E thread:
* The ActiveBundle does not have a registered *bundle finalization handler*. This means the SDK DoFn requests bundle finalization but the runner doesn't support. In x-lang kafka read, it's required by the unbounded sdf wrapper.
* The ActiveBundle does not have a registered bundle *checkpoint handler* This means the SDK DoFn returns a ProcessContinuation and may perform self-checkpoint during execution time but the runner doesn't handle DelayedBundleApplication. In x-lang kafka read, it's required by the sdf Kafka implementation.
In order to use x-lang Kafka with flink runner, we can either support bundle finalization in flink meanwhile use sdf wrapper for Kafka. Or we can support self-checkpoint in flink and use sdf Kafka instead. But in long term, it would be nice to have both in flink runner.
> Flink runner supports Bundle Finalization
> -----------------------------------------
>
> Key: BEAM-6868
> URL: https://issues.apache.org/jira/browse/BEAM-6868
> Project: Beam
> Issue Type: New Feature
> Components: cross-language, runner-flink
> Reporter: Boyuan Zhang
> Priority: P1
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)