You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Abhijeet Kumar <ab...@sentienz.com> on 2018/11/26 12:13:04 UTC
Flink join stream where one stream is coming 5 minutes late
Hello Team,
I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know.
Thanks,
Abhijeet Kumar
Software Development Engineer,
Sentienz Solutions Pvt Ltd
Cognitive Data Platform - Perceive the Data !
abhijeet.kumar@sentienz.com <ma...@sentienz.com> |www.sentienz.com <http://www.sentienz.com/> | Bengaluru
Re: Flink join stream where one stream is coming 5 minutes late
Posted by Abhijeet Kumar <ab...@sentienz.com>.
Thanks for the response, Xingcan. I got your point and the whole idea.
> On 27-Nov-2018, at 12:15 AM, Xingcan Cui <xi...@gmail.com> wrote:
>
> Hi Abhijeet,
>
> If you want to perform window-join in the DataStream API, the window configurations on both sides must be exactly the same.
>
> For your case, maybe you can try adding a 5 mins delay on event times (and watermarks) of the faster stream.
>
> Hope that helps.
>
> Best,
> Xingcan
>
>
>> On Nov 26, 2018, at 12:44 PM, Abhijeet Kumar <abhijeet.kumar@sentienz.com <ma...@sentienz.com>> wrote:
>>
>> I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong.
>>
>>> On 26-Nov-2018, at 5:56 PM, Taher Koitawala <taher.koitawala@gslab.com <ma...@gslab.com>> wrote:
>>>
>>> May I ask why you want to have 2 differences between window time? What's the use case?
>>>
>>> On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <abhijeet.kumar@sentienz.com <ma...@sentienz.com> wrote:
>>> Hello Team,
>>>
>>> I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know.
>>>
>>> Thanks,
>>>
>>>
>>> Abhijeet Kumar
>>> Software Development Engineer,
>>> Sentienz Solutions Pvt Ltd
>>> Cognitive Data Platform - Perceive the Data !
>>> abhijeet.kumar@sentienz.com <ma...@sentienz.com> |www.sentienz.com <http://www.sentienz.com/> | Bengaluru
>>>
>>>
>>
>
Re: Flink join stream where one stream is coming 5 minutes late
Posted by Xingcan Cui <xi...@gmail.com>.
Hi Abhijeet,
If you want to perform window-join in the DataStream API, the window configurations on both sides must be exactly the same.
For your case, maybe you can try adding a 5 mins delay on event times (and watermarks) of the faster stream.
Hope that helps.
Best,
Xingcan
> On Nov 26, 2018, at 12:44 PM, Abhijeet Kumar <ab...@sentienz.com> wrote:
>
> I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong.
>
>> On 26-Nov-2018, at 5:56 PM, Taher Koitawala <taher.koitawala@gslab.com <ma...@gslab.com>> wrote:
>>
>> May I ask why you want to have 2 differences between window time? What's the use case?
>>
>> On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <abhijeet.kumar@sentienz.com <ma...@sentienz.com> wrote:
>> Hello Team,
>>
>> I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know.
>>
>> Thanks,
>>
>>
>> Abhijeet Kumar
>> Software Development Engineer,
>> Sentienz Solutions Pvt Ltd
>> Cognitive Data Platform - Perceive the Data !
>> abhijeet.kumar@sentienz.com <ma...@sentienz.com> |www.sentienz.com <http://www.sentienz.com/> | Bengaluru
>>
>>
>
Re: Flink join stream where one stream is coming 5 minutes late
Posted by Abhijeet Kumar <ab...@sentienz.com>.
I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong.
> On 26-Nov-2018, at 5:56 PM, Taher Koitawala <ta...@gslab.com> wrote:
>
> May I ask why you want to have 2 differences between window time? What's the use case?
>
> On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <abhijeet.kumar@sentienz.com <ma...@sentienz.com> wrote:
> Hello Team,
>
> I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know.
>
> Thanks,
>
>
> Abhijeet Kumar
> Software Development Engineer,
> Sentienz Solutions Pvt Ltd
> Cognitive Data Platform - Perceive the Data !
> abhijeet.kumar@sentienz.com <ma...@sentienz.com> |www.sentienz.com <http://www.sentienz.com/> | Bengaluru
>
>
Re: Flink join stream where one stream is coming 5 minutes late
Posted by Taher Koitawala <ta...@gslab.com>.
May I ask why you want to have 2 differences between window time? What's
the use case?
On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <abhijeet.kumar@sentienz.com
wrote:
> Hello Team,
>
> I've to join two stream where one stream is coming late. So, I planned
> doing it by creating two windows, for first window the size will be 5
> minutes and for the other it will be 10 minutes. Then, I'll perform the
> join operation. Is my solution correct because I don't know whether one
> window will sync with other or not because one is getting finished earlier
> and another is late. If there is a better way to achieve the same, please
> let me know.
>
> Thanks,
>
>
> *Abhijeet Kumar*
> Software Development Engineer,
> Sentienz Solutions Pvt Ltd
> Cognitive Data Platform - Perceive the Data !
> abhijeet.kumar@sentienz.com |www.sentienz.com | Bengaluru
>
>
>