[GitHub] bahir-flink pull request #25: add redis sink checkpoint

[buptljy <gi...@git.apache.org>]

GitHub user buptljy opened a pull request:

    https://github.com/apache/bahir-flink/pull/25

    add redis sink checkpoint

    It seems that the redis sink is not very suitable for some specific circumstances.
    For example, I want to do a word count and I enable the checkpoint machenism, it seems to be wrong if I start my program from checkpoint when my program fails, because datas that already sent will be sent to redisSink again.
    I don't know if it is acceptable for you, so I write a simple version with a flushOnCheckpoint option. I have a further improvement idea that we can use redis pipeline if we are going to store datas into redis when snapshot. 
    I will optimize it if this idea looks good to you.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/buptljy/bahir-flink redissink

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bahir-flink/pull/25.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #25
    
----
commit 683b0ed9ac5665ea731920a92c193589de285d01
Author: Liao Jiayi <li...@ip-192-168-36-150.cn-north-1.compute.internal>
Date:   2017-12-05T03:43:47Z

    add redis sink checkpoint

----


---

[GitHub] bahir-flink pull request #25: add redis sink checkpoint

[buptljy <gi...@git.apache.org>]

Github user buptljy closed the pull request at:

    https://github.com/apache/bahir-flink/pull/25


---

[GitHub] bahir-flink issue #25: add redis sink checkpoint

[rmetzger <gi...@git.apache.org>]

Github user rmetzger commented on the issue:

    https://github.com/apache/bahir-flink/pull/25
  
    Thanks a lot for the pull request. Integrating checkpointing with external connectors is usually a tricky topic. Before we can merge this change, I'd like to fully understand the problem first.
    
    Does the Jedis library buffer writes internally? This would interfere with the checkpointing (like its the case with Flink's Kafka connector), but I don't think that jedis is buffering.
     



---