You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Akshay Agarwal <ak...@grofers.com> on 2021/10/15 08:21:41 UTC

Synchronisation between different sources in flink

Hi Team,

I have a streaming job that creates an enriched stream from streams of 2
different sources like one from files in object storage and another one
Kafka. But since both of these are from different sources, events from them
are read at different moments based on their performance. For example
events from Kafka arrives first than from file storage events and even
though I have used event watermarks, thus at the restart of jobs, enriched
events are wrong and it gets corrected after file events start arriving. I
tried to search for setting up synchronization between different sources in
flink but did not found any blog/material. It might be a noob question, but
if you guys have built something around this, could you let me know.

Regards
*Akshay Agarwal*

-- 
 <https://grofers.com>

Re: Synchronisation between different sources in flink

Posted by Akshay Agarwal <ak...@grofers.com>.
Hey,

The use case is as follows:
Kafka is the source for our frontend/impression events like any user
activity and what we are doing is enriching that stream some values from
the backend which is the source of certain slowly changing properties(that
changes/updated over time) values, it gets dumped as files from the backend
in regular intervals like every hour, quarter of an hour.

*Akshay Agarwal*

On Fri, Oct 15, 2021 at 3:48 PM Martijn Visser <ma...@ververica.com>
wrote:

> Hi,
>
> Can you elaborate a bit more on what your use case is and what you're
> trying to achieve?
>
> Best regards,
>
> Martijn
>
> On Fri, 15 Oct 2021 at 11:25, Akshay Agarwal <ak...@grofers.com>
> wrote:
>
>> Hey,
>>
>> I have few doubts about HybridSource, it says:
>>
>>> To arrange multiple sources in a HybridSource, all sources except the
>>> last one need to be bounded. Therefore, the sources typically need to be
>>> assigned a start and end position.
>>
>> Both sources that I use in the job are unbounded, (file source and Kafka
>> source both get continuous updates). Further schema of both sources is
>> different since they give different data points. So wanted to clarify that
>> further you are suggesting converting Kafka source to hybrid source(file
>> source and Kafka source) and update state based on that and keep file
>> source as it is.
>>
>> *Akshay Agarwal*
>>
>>
>> On Fri, Oct 15, 2021 at 2:06 PM Akshay Agarwal <
>> akshay.agarwal@grofers.com> wrote:
>>
>>> Nope, I haven't gone through that, will take a look, thanks for the
>>> prompt reply.
>>>
>>> *Akshay Agarwal*
>>>
>>>
>>>
>>> On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser <ma...@ververica.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Have you checked out the Hybrid Source? [1]
>>>>
>>>> Thanks, Martijn
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/
>>>>
>>>> On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <
>>>> akshay.agarwal@grofers.com> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I have a streaming job that creates an enriched stream from streams of
>>>>> 2 different sources like one from files in object storage and another one
>>>>> Kafka. But since both of these are from different sources, events from them
>>>>> are read at different moments based on their performance. For example
>>>>> events from Kafka arrives first than from file storage events and even
>>>>> though I have used event watermarks, thus at the restart of jobs, enriched
>>>>> events are wrong and it gets corrected after file events start arriving. I
>>>>> tried to search for setting up synchronization between different sources in
>>>>> flink but did not found any blog/material. It might be a noob question, but
>>>>> if you guys have built something around this, could you let me know.
>>>>>
>>>>> Regards
>>>>> *Akshay Agarwal*
>>>>>
>>>>> [image: https://grofers.com] <https://grofers.com>
>>>>
>>>>
>> [image: https://grofers.com] <https://grofers.com>
>
>

-- 
 <https://grofers.com>

Re: Synchronisation between different sources in flink

Posted by Martijn Visser <ma...@ververica.com>.
Hi,

Can you elaborate a bit more on what your use case is and what you're
trying to achieve?

Best regards,

Martijn

On Fri, 15 Oct 2021 at 11:25, Akshay Agarwal <ak...@grofers.com>
wrote:

> Hey,
>
> I have few doubts about HybridSource, it says:
>
>> To arrange multiple sources in a HybridSource, all sources except the
>> last one need to be bounded. Therefore, the sources typically need to be
>> assigned a start and end position.
>
> Both sources that I use in the job are unbounded, (file source and Kafka
> source both get continuous updates). Further schema of both sources is
> different since they give different data points. So wanted to clarify that
> further you are suggesting converting Kafka source to hybrid source(file
> source and Kafka source) and update state based on that and keep file
> source as it is.
>
> *Akshay Agarwal*
>
>
> On Fri, Oct 15, 2021 at 2:06 PM Akshay Agarwal <ak...@grofers.com>
> wrote:
>
>> Nope, I haven't gone through that, will take a look, thanks for the
>> prompt reply.
>>
>> *Akshay Agarwal*
>>
>>
>>
>> On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser <ma...@ververica.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Have you checked out the Hybrid Source? [1]
>>>
>>> Thanks, Martijn
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/
>>>
>>> On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <ak...@grofers.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I have a streaming job that creates an enriched stream from streams of
>>>> 2 different sources like one from files in object storage and another one
>>>> Kafka. But since both of these are from different sources, events from them
>>>> are read at different moments based on their performance. For example
>>>> events from Kafka arrives first than from file storage events and even
>>>> though I have used event watermarks, thus at the restart of jobs, enriched
>>>> events are wrong and it gets corrected after file events start arriving. I
>>>> tried to search for setting up synchronization between different sources in
>>>> flink but did not found any blog/material. It might be a noob question, but
>>>> if you guys have built something around this, could you let me know.
>>>>
>>>> Regards
>>>> *Akshay Agarwal*
>>>>
>>>> [image: https://grofers.com] <https://grofers.com>
>>>
>>>
> [image: https://grofers.com] <https://grofers.com>

Re: Synchronisation between different sources in flink

Posted by Akshay Agarwal <ak...@grofers.com>.
Hey,

I have few doubts about HybridSource, it says:

> To arrange multiple sources in a HybridSource, all sources except the
> last one need to be bounded. Therefore, the sources typically need to be
> assigned a start and end position.

Both sources that I use in the job are unbounded, (file source and Kafka
source both get continuous updates). Further schema of both sources is
different since they give different data points. So wanted to clarify that
further you are suggesting converting Kafka source to hybrid source(file
source and Kafka source) and update state based on that and keep file
source as it is.

*Akshay Agarwal*


On Fri, Oct 15, 2021 at 2:06 PM Akshay Agarwal <ak...@grofers.com>
wrote:

> Nope, I haven't gone through that, will take a look, thanks for the prompt
> reply.
>
> *Akshay Agarwal*
>
>
>
> On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser <ma...@ververica.com>
> wrote:
>
>> Hi,
>>
>> Have you checked out the Hybrid Source? [1]
>>
>> Thanks, Martijn
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/
>>
>> On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <ak...@grofers.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> I have a streaming job that creates an enriched stream from streams of 2
>>> different sources like one from files in object storage and another one
>>> Kafka. But since both of these are from different sources, events from them
>>> are read at different moments based on their performance. For example
>>> events from Kafka arrives first than from file storage events and even
>>> though I have used event watermarks, thus at the restart of jobs, enriched
>>> events are wrong and it gets corrected after file events start arriving. I
>>> tried to search for setting up synchronization between different sources in
>>> flink but did not found any blog/material. It might be a noob question, but
>>> if you guys have built something around this, could you let me know.
>>>
>>> Regards
>>> *Akshay Agarwal*
>>>
>>> [image: https://grofers.com] <https://grofers.com>
>>
>>

-- 
 <https://grofers.com>

Re: Synchronisation between different sources in flink

Posted by Akshay Agarwal <ak...@grofers.com>.
Nope, I haven't gone through that, will take a look, thanks for the prompt
reply.

*Akshay Agarwal*



On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser <ma...@ververica.com>
wrote:

> Hi,
>
> Have you checked out the Hybrid Source? [1]
>
> Thanks, Martijn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/
>
> On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <ak...@grofers.com>
> wrote:
>
>> Hi Team,
>>
>> I have a streaming job that creates an enriched stream from streams of 2
>> different sources like one from files in object storage and another one
>> Kafka. But since both of these are from different sources, events from them
>> are read at different moments based on their performance. For example
>> events from Kafka arrives first than from file storage events and even
>> though I have used event watermarks, thus at the restart of jobs, enriched
>> events are wrong and it gets corrected after file events start arriving. I
>> tried to search for setting up synchronization between different sources in
>> flink but did not found any blog/material. It might be a noob question, but
>> if you guys have built something around this, could you let me know.
>>
>> Regards
>> *Akshay Agarwal*
>>
>> [image: https://grofers.com] <https://grofers.com>
>
>

-- 
 <https://grofers.com>

Re: Synchronisation between different sources in flink

Posted by Martijn Visser <ma...@ververica.com>.
Hi,

Have you checked out the Hybrid Source? [1]

Thanks, Martijn

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/

On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <ak...@grofers.com>
wrote:

> Hi Team,
>
> I have a streaming job that creates an enriched stream from streams of 2
> different sources like one from files in object storage and another one
> Kafka. But since both of these are from different sources, events from them
> are read at different moments based on their performance. For example
> events from Kafka arrives first than from file storage events and even
> though I have used event watermarks, thus at the restart of jobs, enriched
> events are wrong and it gets corrected after file events start arriving. I
> tried to search for setting up synchronization between different sources in
> flink but did not found any blog/material. It might be a noob question, but
> if you guys have built something around this, could you let me know.
>
> Regards
> *Akshay Agarwal*
>
> [image: https://grofers.com] <https://grofers.com>