You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jin Yi <ji...@promoted.ai> on 2021/08/17 01:15:26 UTC
redis sink from flink
is apache bahir still a thing? it hasn't been touched for months (since
redis 2.8.5).
as such, looking at the current flink connector docs, it's no longer
pointing to anything from the bahir project. looking around in either the
flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL.
is the best bet for a flink to redis sink something i roll on my own
(inclined to go this route w/ buffered writes)? or should i try going
through via kafka and using confluent's kafka redis connector (flink =>
kafka => redis)?
Re: redis sink from flink
Posted by Jin Yi <ji...@promoted.ai>.
great, thanks for the pointers everyone.
i'm going to pursue rolling my own built around lettuce since it seems more
feature-full wrt async semantics.
On Mon, Aug 16, 2021 at 7:21 PM Yik San Chan <ev...@gmail.com>
wrote:
> By the way, this post in Chinese showed how we do it exactly with code.
>
> https://yiksanchan.com/posts/flink-bulk-insert-redis
>
> And yes it had buffered writes support by leveraging Flink operator state,
> and Redis Pipelining. Feel free to let you know if you have any questions.
>
>
> On Tue, Aug 17, 2021 at 10:15 AM Yik San Chan <ev...@gmail.com>
> wrote:
>
>> Hi Jin,
>>
>> I was in the same shoes. I tried bahir redis connector at first, then I
>> felt it was very limited, so I rolled out my own. It was actually quite
>> straightforward.
>>
>> All you need to do is to extend RichSinkFunction, then put your logic
>> inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis)
>> is quite popular and simple to get started.
>>
>> Let me know if you love to learn more details about our implementation.
>>
>> Best,
>> Yik San
>>
>> On Tue, Aug 17, 2021 at 9:15 AM Jin Yi <ji...@promoted.ai> wrote:
>>
>>> is apache bahir still a thing? it hasn't been touched for months (since
>>> redis 2.8.5).
>>>
>>> as such, looking at the current flink connector docs, it's no longer
>>> pointing to anything from the bahir project. looking around in either the
>>> flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL.
>>>
>>> is the best bet for a flink to redis sink something i roll on my own
>>> (inclined to go this route w/ buffered writes)? or should i try going
>>> through via kafka and using confluent's kafka redis connector (flink =>
>>> kafka => redis)?
>>>
>>
Re: redis sink from flink
Posted by Yik San Chan <ev...@gmail.com>.
By the way, this post in Chinese showed how we do it exactly with code.
https://yiksanchan.com/posts/flink-bulk-insert-redis
And yes it had buffered writes support by leveraging Flink operator state,
and Redis Pipelining. Feel free to let you know if you have any questions.
On Tue, Aug 17, 2021 at 10:15 AM Yik San Chan <ev...@gmail.com>
wrote:
> Hi Jin,
>
> I was in the same shoes. I tried bahir redis connector at first, then I
> felt it was very limited, so I rolled out my own. It was actually quite
> straightforward.
>
> All you need to do is to extend RichSinkFunction, then put your logic
> inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis)
> is quite popular and simple to get started.
>
> Let me know if you love to learn more details about our implementation.
>
> Best,
> Yik San
>
> On Tue, Aug 17, 2021 at 9:15 AM Jin Yi <ji...@promoted.ai> wrote:
>
>> is apache bahir still a thing? it hasn't been touched for months (since
>> redis 2.8.5).
>>
>> as such, looking at the current flink connector docs, it's no longer
>> pointing to anything from the bahir project. looking around in either the
>> flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL.
>>
>> is the best bet for a flink to redis sink something i roll on my own
>> (inclined to go this route w/ buffered writes)? or should i try going
>> through via kafka and using confluent's kafka redis connector (flink =>
>> kafka => redis)?
>>
>
Re: redis sink from flink
Posted by Yik San Chan <ev...@gmail.com>.
Hi Jin,
I was in the same shoes. I tried bahir redis connector at first, then I
felt it was very limited, so I rolled out my own. It was actually quite
straightforward.
All you need to do is to extend RichSinkFunction, then put your logic
inside. Regarding Redis clients, Jedis (https://github.com/redis/jedis) is
quite popular and simple to get started.
Let me know if you love to learn more details about our implementation.
Best,
Yik San
On Tue, Aug 17, 2021 at 9:15 AM Jin Yi <ji...@promoted.ai> wrote:
> is apache bahir still a thing? it hasn't been touched for months (since
> redis 2.8.5).
>
> as such, looking at the current flink connector docs, it's no longer
> pointing to anything from the bahir project. looking around in either the
> flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL.
>
> is the best bet for a flink to redis sink something i roll on my own
> (inclined to go this route w/ buffered writes)? or should i try going
> through via kafka and using confluent's kafka redis connector (flink =>
> kafka => redis)?
>
Re: redis sink from flink
Posted by Yangze Guo <ka...@gmail.com>.
Hi, Jin
IIUC, the DataStream connector `RedisSink` can still be used. However,
the Table API connector `RedisTableSink` might not work (at least in
the future) because it is implemented based on the deprecated Table
connector abstraction. You can still give it a try, though.
Best,
Yangze Guo
On Tue, Aug 17, 2021 at 9:15 AM Jin Yi <ji...@promoted.ai> wrote:
>
> is apache bahir still a thing? it hasn't been touched for months (since redis 2.8.5).
>
> as such, looking at the current flink connector docs, it's no longer pointing to anything from the bahir project. looking around in either the flink or bahir newsgroups doesn't turn up anything regarding bahir's EOL.
>
> is the best bet for a flink to redis sink something i roll on my own (inclined to go this route w/ buffered writes)? or should i try going through via kafka and using confluent's kafka redis connector (flink => kafka => redis)?