You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/22 23:35:52 UTC
[GitHub] [incubator-pinot] fx19880617 opened a new issue #6480: Support Kafka Consumer in exact-once semantics
fx19880617 opened a new issue #6480:
URL: https://github.com/apache/incubator-pinot/issues/6480
Seeing more requirements on this.
Kafka consumer supports exact-once semantics but requires code changes.
Code snippet:
```
KafkaConsumer consumer = createKafkaConsumer(
"bootstrap.servers", "localhost:9092",
"group.id", "my-group-id",
"isolation.level", "read_committed");
consumer.subscribe(singleton("inputTopic"));
while (true) {
ConsumerRecords records = consumer.poll(Long.MAX_VALUE);
producer.beginTransaction();
for (ConsumerRecord record : records)
producer.send(producerRecord("outputTopic", record));
producer.sendOffsetsToTransaction(currentOffsets(consumer), group);
producer.commitTransaction();
}
```
We can add a config in kafka-2.0 plugin and switch the code path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #6480: Support Kafka Consumer in exact-once semantics
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6480:
URL: https://github.com/apache/incubator-pinot/issues/6480#issuecomment-782466167
@Jackie-Jiang plz review it at https://github.com/pinot-contrib/pinot-docs/pull/24
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6480: Support Kafka Consumer in exact-once semantics
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6480:
URL: https://github.com/apache/incubator-pinot/issues/6480#issuecomment-779547490
@yupeng9 Thanks for adding the support. Can you please also add it in the documentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6480: Support Kafka Consumer in exact-once semantics
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6480:
URL: https://github.com/apache/incubator-pinot/issues/6480#issuecomment-790110992
Supported in PR #6580. This feature will be available in Pinot 0.7.0 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 closed issue #6480: Support Kafka Consumer in exact-once semantics
Posted by GitBox <gi...@apache.org>.
fx19880617 closed issue #6480:
URL: https://github.com/apache/incubator-pinot/issues/6480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org