You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Yufan Sheng (Jira)" <ji...@apache.org> on 2022/12/14 09:50:00 UTC
[jira] [Created] (FLINK-30413) Drop Share and Key_Shared subscription support in Pulsar connector
Yufan Sheng created FLINK-30413:
-----------------------------------
Summary: Drop Share and Key_Shared subscription support in Pulsar connector
Key: FLINK-30413
URL: https://issues.apache.org/jira/browse/FLINK-30413
Project: Flink
Issue Type: Improvement
Components: Connectors / Pulsar
Affects Versions: 1.17.0
Reporter: Yufan Sheng
Fix For: 1.17.0
A lot of Pulsar connector test unstable issues are related to {{Shared}} and {{Key_Shared}} subscription. Because this two subscription is designed to consume the records in an unordered way. And we can support multiple consumers in same topic partition. But this feature lead to some drawbacks in connector.
1. Performance
Flink is a true stream processor with high correctness support. But support multiple consumer will require higher correctness which depends on Pulsar transaction. But the internal implementation of Pulsar transaction on source is record the message one by one and stores all the pending ack status in client side. Which is slow and memory inefficient.
This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with low throughput. This against our intention to support these two subscription. Because adding multiple consumer to same partition can increase the consuming speed.
2. Unstable
Pulsar transaction acknowledge the messages one by one in an internal Pulsar's topic
--
This message was sent by Atlassian Jira
(v8.20.10#820010)