[jira] [Comment Edited] (FLINK-2624) RabbitMQ source / sink should participate in checkpointing

["Hilmi Yildirim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733814#comment-14733814 ] 

Hilmi Yildirim edited comment on FLINK-2624 at 9/7/15 3:05 PM:
---------------------------------------------------------------

Unfortunately, there is another problem. RabbitMQ delivers messages to the consumers and deletes messages if they are acknowledged. RabbitMQ does deliver a single message to only one consumer and waits for the acknowledgement until the connection is lost, e.g. the consumer crashes. In this case RabbitMQ sends the messages which are not acknowledged to other consumsers. For example, consumer c received messages m1,m2 and m3. If a consumer has acknowledged only m1 without m2 and m3 and he crashes, then m2 and m3 will be delivered to other conumsers.

This means in our case: If a worker node crashes during the snapshotting without acknowledging all processed messages, then the state of this worker node is useless. Furthermore, the messages the consumer has acknowledged before he crashed, are deleted.


was (Author: hilmiyildirim):
Unfortunately, there is another problem. RabbitMQ delivers messages to the consumers and deletes messages if they are acknowledged. RabbitMQ does deliver a message to only one consumer and waits for the acknowledgement until the connection is lost, e.g. the consumer crashes. In this case RabbitMQ sends the messages which are not acknowledged to other consumsers. For example, consumer c received messages m1,m2 and m3. If a consumer has acknowledged only m1 without m2 and m3 and he crashes, then m2 and m3 will be delivered to other conumsers.

This means in our case: If a worker node crashes during the snapshotting without acknowledging all processed messages, then the state of this worker node is useless. Furthermore, the messages the consumer has acknowledged before he crashed, are deleted.

> RabbitMQ source / sink should participate in checkpointing
> ----------------------------------------------------------
>
>                 Key: FLINK-2624
>                 URL: https://issues.apache.org/jira/browse/FLINK-2624
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Hilmi Yildirim
>
> The RabbitMQ connector does not offer any fault tolerance guarantees right now, because it does not participate in the checkpointing.
> We should integrate it in a similar was as the {{FlinkKafkaConsumer}} is integrated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)