You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Stephan Ewen (JIRA)" <ji...@apache.org> on 2015/09/22 13:30:04 UTC
[jira] [Created] (FLINK-2727) Add a base class for
MessageQueue-with-acknowledgement sources
Stephan Ewen created FLINK-2727:
-----------------------------------
Summary: Add a base class for MessageQueue-with-acknowledgement sources
Key: FLINK-2727
URL: https://issues.apache.org/jira/browse/FLINK-2727
Project: Flink
Issue Type: New Feature
Components: Streaming
Affects Versions: 0.10
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Several message queues (RabbitMQ, Amazon SQS) have the pattern that you retrieve messages and acknowledge them back by ID.
We can create a simple base non-parallel source that provides tooling for:
- Collecting the IDs of elements emitted between two checkpoints
- Persisting them with the checkpoint, respecting proper serialization
- Acknowledging them when a checkpoint is notified of completion.
This assumes that the Message Queues retain unacknowledged messages and re-emit them after the acknowledgement period expired.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)