You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Weiqing Yang <ya...@gmail.com> on 2024/12/23 07:19:43 UTC
[DISCUSS] Early Fire Support for Flink SQL Interval Join
Hi all,
I’d like to initiate a discussion about introducing early fire support for
Flink SQL interval joins.
*Motivation*
In many streaming applications, particularly real-time analytics and
monitoring systems, it is valuable to obtain partial results earlier rather
than waiting for full join conditions to finalize. For Flink SQL interval
joins, results are typically delayed until watermarks ensure no more
matches can occur. This delay can be challenging for scenarios that require
fast feedback. Early fire support addresses this by emitting intermediate
results speculatively and using retractions or updates to maintain eventual
consistency and ensure correctness.
Here’s the proposal document: Link
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph>
Your feedback and ideas are welcome to refine this feature.
Thanks,
Weiqing
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Adding @Yuepeng @Martijn
On Mon, Mar 10, 2025 at 7:36 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi Becket,
>
> Thanks for the feedback! I’ve updated the design doc with a new
> “Granularity of the Early Fire Hint” section: link
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.gle8q83bjlj5>
> .
>
> Here’s a quick summary:
>
> - EARLY_FIRE is scoped to a single operator. If multiple operators in
> one query can support early firing, Flink applies the hint to the first
> matching operator and logs a warning to highlight the ambiguity. Subsequent
> operators will not receive the hint.
> - Split queries for different early-fire settings. If you need
> separate early-fire configurations - or just want to avoid ambiguity -
> splitting the SQL into multiple statements or views ensures each operator
> has its own distinct EARLY_FIRE hint.
>
> Please let me know if you have any other questions or suggestions.
>
> Best,
> Weiqing
>
> On Fri, Feb 28, 2025 at 5:01 PM Becket Qin <be...@gmail.com> wrote:
>
>> Thanks for updating the FLIP, Weiqing.
>>
>> Another question, what is the granularity of the early fire hint? Will the
>> hint be applied to all the outer joins in the same block? What if in the
>> query block there are also aggregations? For example:
>>
>> SELECT window_start, window_end, buyer, count(*) /*+
>> EARLY_FIRE('delay'='5s'
>> , 'time_mode'='rowtime|proctime') */
>> FROM Orders o LEFT JOIN Shipments s
>> WHERE o.id = s.order_id
>> AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
>> GROUP BY o.window_start, o.window_end, buyer;
>>
>> What will be the behavior? Will both the aggregation and interval join? Or
>> will it only be applied to the join?
>> It would be useful to define the behavior here.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <ya...@gmail.com>
>> wrote:
>>
>> > Thanks for the feedback, Becket. I’ve heard similar concerns from other
>> > reviewers, and I now agree that the `interval` option is likely not
>> widely
>> > used. In the FLIP, it was intended specifically for Interval Join, not
>> as a
>> > general early-fire mechanism. We can consider making it more generic in
>> a
>> > future update. Since it’s optional, I’ll go ahead and remove it for
>> > clarity.
>> >
>> > Appreciate your input!
>> >
>> > Best,
>> > Weiqing
>> >
>> > On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <be...@gmail.com>
>> wrote:
>> >
>> > > Sorry for the late reply. I have a question regarding the
>> configuration
>> > of
>> > > interval. I am wondering in which scenario the interval config will
>> > > actually be used. Is the interval configuration only useful for
>> windowed
>> > > aggregation?
>> > >
>> > > For the join operations, early fire seems only applicable to the outer
>> > join
>> > > before there is a match. Once there is a match, the output of the join
>> > > operation should be just event driven. For example, consider the
>> > following
>> > > scenario:
>> > > 1. left outer join, and there is no match from the right side before
>> the
>> > > initial delay
>> > > 2. a record [left, null] was emitted due to early fire
>> > > 3. one of the following two cases must happen
>> > > a. there is still no match event from the right side before the
>> > > specific interval passes. In this case, we are not supposed to emit
>> > another
>> > > [left, null].
>> > > b. if there is a match, a record of [left, right] should be
>> emitted
>> > > immediately regardless of the interval. If this happens, the interval
>> > will
>> > > be ignored from this point on.
>> > >
>> > > That said, if the configuration proposed in this FLIP is intended not
>> > only
>> > > for join, but for general purpose early fire, then the interval config
>> > > makes sense.
>> > >
>> > > Thanks,
>> > >
>> > > Jiangjie (Becket) Qin
>> > >
>> > > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <
>> yangweiqing001@gmail.com>
>> > > wrote:
>> > >
>> > > > Thank you all for reviewing. As there are no objections, I’ll move
>> > > forward
>> > > > with a vote.
>> > > >
>> > > > Best regards,
>> > > > Weiqing
>> > > >
>> > > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
>> > > > vsowrira@asu.edu>
>> > > > wrote:
>> > > >
>> > > > > Same here, I don't have any more questions. Thanks!
>> > > > >
>> > > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > Hi Weiqing,
>> > > > > >
>> > > > > > I don't have any more questions. The doc looks good to me.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Xingcan
>> > > > > >
>> > > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
>> > > > > > vsowrira@asu.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Weiqing,
>> > > > > > >
>> > > > > > > Thanks, that makes sense! Looks like I missed it.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > > Venkata krishnan
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
>> > > > > yangweiqing001@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Venkata,
>> > > > > > > >
>> > > > > > > > * > where only one earlyFire is fired The DELAY *
>> > > > > > > >
>> > > > > > > > The DELAY option mentioned in the Public Interfaces section
>> > > > > > > > <
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
>> > > > > > > > >
>> > > > > > > > of the proposal can produce a single early fire per
>> interval,
>> > > > > aligning
>> > > > > > > with
>> > > > > > > > your suggestion. How it integrates with existing early-fire
>> > > > > > > configurations
>> > > > > > > > are mentioned here
>> > > > > > > > <
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
>> > > > > > > > >.
>> > > > > > > > Let me know if you have any further questions or feedback!
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Weiqing
>> > > > > > > >
>> > > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan
>> <
>> > > > > > > > vsowrira@asu.edu> wrote:
>> > > > > > > >
>> > > > > > > > > Thanks for the response!
>> > > > > > > > >
>> > > > > > > > > > If the optional `interval` in the proposal is enabled,
>> > > > early-fire
>> > > > > > > > outputs
>> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just
>> > > once.
>> > > > > > After
>> > > > > > > > that
>> > > > > > > > > interval concludes, there will also be a final emission.
>> > > > > > > > >
>> > > > > > > > > One more question, would it make sense to support a
>> mechanism
>> > > > > through
>> > > > > > > > > configuration where only one earlyFire is fired for an
>> > interval
>> > > > or
>> > > > > > > > window?
>> > > > > > > > > This would also simplify the state overhead for use cases
>> > where
>> > > > > only
>> > > > > > > one
>> > > > > > > > > early fire is required within a window or interval.
>> Thoughts?
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Venkata krishnan
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
>> > > > > > yangweiqing001@gmail.com
>> > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Venkata,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for your feedback!
>> > > > > > > > > >
>> > > > > > > > > > If the optional `interval` in the proposal is enabled,
>> > > > early-fire
>> > > > > > > > outputs
>> > > > > > > > > > will occur repeatedly at every earlyFireInterval, not
>> just
>> > > > once.
>> > > > > > > After
>> > > > > > > > > that
>> > > > > > > > > > interval concludes, there will also be a final emission.
>> > > > > > > > > >
>> > > > > > > > > > We haven’t included a late-fire mechanism for interval
>> > joins
>> > > in
>> > > > > > this
>> > > > > > > > > FLIP,
>> > > > > > > > > > but it’s certainly something we can consider for future
>> > > > efforts!
>> > > > > > > > > >
>> > > > > > > > > > Best regards,
>> > > > > > > > > > Weiqing
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan
>> Sowrirajan
>> > <
>> > > > > > > > > > vsowrira@asu.edu>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Weiqing,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also
>> > instead
>> > > > of
>> > > > > > > > > > controlling
>> > > > > > > > > > > and configuring through Flink configs unlike the older
>> > > window
>> > > > > > > > > > aggregation,
>> > > > > > > > > > > hints seems to be a much better approach. This
>> enables a
>> > > > > > > customizable
>> > > > > > > > > > early
>> > > > > > > > > > > fire behavior for individual interval joins.
>> > > > > > > > > > >
>> > > > > > > > > > > Couple of questions:
>> > > > > > > > > > >
>> > > > > > > > > > > 1. Does the *early fire* emit an output every
>> > > > earlyFireInterval
>> > > > > > > time
>> > > > > > > > or
>> > > > > > > > > > > will it be a one time output emission and another
>> output
>> > > > > emitted
>> > > > > > at
>> > > > > > > > the
>> > > > > > > > > > end
>> > > > > > > > > > > of the interval?
>> > > > > > > > > > > 2. Are there plans to support *late fire *similar to
>> the
>> > > > window
>> > > > > > > > > > > aggregations in later FLIPs?
>> > > > > > > > > > >
>> > > > > > > > > > > Regards
>> > > > > > > > > > > Venkata krishnan
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
>> > > > > > > > yangweiqing001@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Thanks for reviewing, Xuyang!
>> > > > > > > > > > > >
>> > > > > > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any
>> > > concerns?
>> > > > > > > > > > > >
>> > > > > > > > > > > > If no further objections arise from anyone, I’ll
>> > proceed
>> > > to
>> > > > > > mark
>> > > > > > > > FLIP
>> > > > > > > > > > as
>> > > > > > > > > > > > ready for voting.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > Weiqing
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
>> > > xyzhong131@163.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > LGTM overall. Thanks for updating. I have no
>> problem
>> > > and
>> > > > +1
>> > > > > > for
>> > > > > > > > > this
>> > > > > > > > > > > > > feature.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > --
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best！
>> > > > > > > > > > > > > Xuyang
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
>> > > > > > yangweiqing001@gmail.com
>> > > > > > > >
>> > > > > > > > > 写道：
>> > > > > > > > > > > > > >Hi Xuyang,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >Thank you for your detailed feedback! I’ve
>> updated
>> > the
>> > > > > > > proposal
>> > > > > > > > > doc
>> > > > > > > > > > > > > >accordingly. Please feel free to take another
>> look
>> > and
>> > > > let
>> > > > > > me
>> > > > > > > > know
>> > > > > > > > > > if
>> > > > > > > > > > > > you
>> > > > > > > > > > > > > >have any further thoughts or suggestions.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >Best regards,
>> > > > > > > > > > > > > >Weiqing
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
>> > > > > xyzhong131@163.com>
>> > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >> Hi, Weiqing.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> After reading the new FLIP, I have no issues
>> with
>> > > the
>> > > > > part
>> > > > > > > > > `public
>> > > > > > > > > > > > > >> interface`. I only have some questions
>> regarding
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> the details in the Proposed Changes section.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
>> > > > > > IntervalJoin
>> > > > > > > > > node,
>> > > > > > > > > > > > IIUC:
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind
>> of
>> > > the
>> > > > > > > > > IntervalJoin
>> > > > > > > > > > > can
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending
>> entirely on
>> > > the
>> > > > > > > > > > requirements
>> > > > > > > > > > > of
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> - When early firing is disabled, the
>> UpdateKind of
>> > > the
>> > > > > > > > > > IntervalJoin
>> > > > > > > > > > > is
>> > > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> - Nevertheless, whether early firing is
>> enabled or
>> > > > > > disabled,
>> > > > > > > > the
>> > > > > > > > > > > > > >> IntervalJoin should always require its input to
>> > keep
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
>> > > NONE.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Best！
>> > > > > > > > > > > > > >> Xuyang
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
>> > > > > > > > > yangweiqing001@gmail.com>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >> >Hi Xingcan and Xuyang,
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >Thanks so much for the feedback - it was very
>> > > > helpful!
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >*> 1. The current output stream of a time
>> > interval
>> > > > > outer
>> > > > > > > join
>> > > > > > > > > is
>> > > > > > > > > > an
>> > > > > > > > > > > > > >> >append-only stream. This change will make it a
>> > > > > potential
>> > > > > > > > > > > retractable
>> > > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
>> > > > dynamic
>> > > > > > > output
>> > > > > > > > > > type
>> > > > > > > > > > > > like
>> > > > > > > > > > > > > >> >that. Could you add this part to your design
>> > doc?*
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > - Yes, enabling early firing on time
>> interval
>> > > > outer
>> > > > > > > joins
>> > > > > > > > > can
>> > > > > > > > > > > emit
>> > > > > > > > > > > > > >> > retractions when previously emitted rows
>> are
>> > > > updated
>> > > > > > or
>> > > > > > > > > > > > invalidated
>> > > > > > > > > > > > > by
>> > > > > > > > > > > > > >> > later matches. I’ve updated the proposal
>> > > (Planner
>> > > > > > > > Awareness
>> > > > > > > > > > > > > >> > <
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > and Changes in
>> > > FlinkChangelogModeInferenceProgram
>> > > > > > > > > > > > > >> > <
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
>> > > > > > > > > > > > > >> >)
>> > > > > > > > > > > > > >> > to clarify that the stream might switch
>> from
>> > > > > > append-only
>> > > > > > > > to
>> > > > > > > > > a
>> > > > > > > > > > > > > >> > retract/upsert stream. Let me know if
>> anything
>> > > is
>> > > > > > > missing.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >*> 2. What's the use case when the downstream
>> > > > > components
>> > > > > > > need
>> > > > > > > > > to
>> > > > > > > > > > > get
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > >> >early fired results regularly?*
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > - The new INTERVAL option (in addition to
>> > DELAY)
>> > > > > > allows
>> > > > > > > > > > periodic
>> > > > > > > > > > > > > >> updates
>> > > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
>> > > delay.
>> > > > > This
>> > > > > > > > > > captures
>> > > > > > > > > > > > how
>> > > > > > > > > > > > > >> results
>> > > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
>> > > > > > “Repeatedly”
>> > > > > > > > > > option.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >*> 3. The time interval join operator itself
>> is
>> > not
>> > > > > quite
>> > > > > > > > > > efficient
>> > > > > > > > > > > > > when
>> > > > > > > > > > > > > >> >the state becomes large. Will there be any
>> extra
>> > > > > overhead
>> > > > > > > > after
>> > > > > > > > > > > > > >> introducing
>> > > > > > > > > > > > > >> >this feature?*
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > - Early fire does introduce some overhead
>> by
>> > > > > > potentially
>> > > > > > > > > > > emitting
>> > > > > > > > > > > > > >> > partial matches multiple times with
>> retraction
>> > > > > > (avoiding
>> > > > > > > > > > > duplicate
>> > > > > > > > > > > > > >> outputs
>> > > > > > > > > > > > > >> > though). However, if it’s disabled, there
>> is
>> > no
>> > > > > > > additional
>> > > > > > > > > > cost.
>> > > > > > > > > > > > > Most
>> > > > > > > > > > > > > >> users
>> > > > > > > > > > > > > >> > find the performance trade-off acceptable
>> for
>> > > the
>> > > > > > > > real-time
>> > > > > > > > > > > > > insights it
>> > > > > > > > > > > > > >> > provides.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >*> 1. Currently, there are some configs
>> related
>> > to
>> > > > > early
>> > > > > > > > firing
>> > > > > > > > > > > > > available
>> > > > > > > > > > > > > >> >to users:
>> `table.exec.emit.early-fire.en**abled`
>> > > and
>> > > > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
>> > > > > > > > > > > > > >> >**lay`.
>> > > > > > > > > > > > > >> >Although their documentation states that they
>> are
>> > > > only
>> > > > > > > > > applicable
>> > > > > > > > > > > to
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > >> >Window operator, it seems possible that they
>> may
>> > > also
>> > > > > be
>> > > > > > > > > relevant
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > >> >context of this FLIP. Otherwise, having
>> different
>> > > > early
>> > > > > > > > firing
>> > > > > > > > > > > > > behaviors
>> > > > > > > > > > > > > >> >for different operators could confuse users.*
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to
>> avoid
>> > > > > > > confusion.
>> > > > > > > > > I’ve
>> > > > > > > > > > > > > added a
>> > > > > > > > > > > > > >> > section
>> > > > > > > > > > > > > >> > <
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >in
>> > > > > > > > > > > > > >> > the proposal highlighting that we should
>> align
>> > > > > > > hint-based
>> > > > > > > > > > > interval
>> > > > > > > > > > > > > join
>> > > > > > > > > > > > > >> > configurations with the existing
>> > > table.exec.emit.*
>> > > > > > > > settings.
>> > > > > > > > > > > > > >> Suggestions
>> > > > > > > > > > > > > >> > on how to make the unification are
>> welcome! We
>> > > > plan
>> > > > > to
>> > > > > > > > > extend
>> > > > > > > > > > > > early
>> > > > > > > > > > > > > >> firing
>> > > > > > > > > > > > > >> > to window joins via hints in a future FLIP.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
>> > > Similar
>> > > > > to
>> > > > > > > > point
>> > > > > > > > > 1,
>> > > > > > > > > > > > > perhaps
>> > > > > > > > > > > > > >> >we can introduce it to other window-related
>> > > operators
>> > > > > in
>> > > > > > > the
>> > > > > > > > > > > future.>
>> > > > > > > > > > > > > 3.
>> > > > > > > > > > > > > >> >You need to modify the
>> > > > > FlinkChangelogModeInferenceProgram
>> > > > > > > to
>> > > > > > > > > > ensure
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > >> >downstream nodes of interval joins with early
>> > > firing
>> > > > > > > enabled
>> > > > > > > > > are
>> > > > > > > > > > > > aware
>> > > > > > > > > > > > > of
>> > > > > > > > > > > > > >> >retract or upsert messages.*
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > - We agree that time_mode could be
>> introduced
>> > to
>> > > > > other
>> > > > > > > > > > > > window-based
>> > > > > > > > > > > > > >> > operators down the road. We also want to
>> > support
>> > > > > early
>> > > > > > > > fire
>> > > > > > > > > > for
>> > > > > > > > > > > > > >> > window join. Also, thanks for highlighting
>> > > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added
>> > the
>> > > > code
>> > > > > > > > change
>> > > > > > > > > on
>> > > > > > > > > > > it
>> > > > > > > > > > > > > >> > <
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > in the proposal.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
>> > > updated
>> > > > > the
>> > > > > > > > > > proposal
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > >> >these points. Please let me know if there’s
>> > > anything
>> > > > > > else I
>> > > > > > > > > > should
>> > > > > > > > > > > > > >> address.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >Best,
>> > > > > > > > > > > > > >> >Weiqing
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
>> > > > > > xyzhong131@163.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this
>> FLIP.
>> > I
>> > > > > have a
>> > > > > > > few
>> > > > > > > > > > > > > questions:
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> 1. Currently, there are some configs
>> related to
>> > > > early
>> > > > > > > > firing
>> > > > > > > > > > > > > available
>> > > > > > > > > > > > > >> to
>> > > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled`
>> and
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
>> > > their
>> > > > > > > > > > documentation
>> > > > > > > > > > > > > states
>> > > > > > > > > > > > > >> >> that they are only applicable to the Window
>> > > > operator,
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> it seems possible that they may also be
>> > relevant
>> > > in
>> > > > > the
>> > > > > > > > > context
>> > > > > > > > > > > of
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > >> >> FLIP. Otherwise, having different early
>> firing
>> > > > > > behaviors
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> for different operators could confuse users.
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
>> > > Similar
>> > > > to
>> > > > > > > point
>> > > > > > > > > 1,
>> > > > > > > > > > > > > perhaps
>> > > > > > > > > > > > > >> we
>> > > > > > > > > > > > > >> >> can introduce it to other window-related
>> > > operators
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> in the future.
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> 3. You need to modify the
>> > > > > > > > FlinkChangelogModeInferenceProgram
>> > > > > > > > > to
>> > > > > > > > > > > > > ensure
>> > > > > > > > > > > > > >> >> that downstream nodes of interval joins with
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> early firing enabled are aware of retract or
>> > > upsert
>> > > > > > > > messages.
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> --
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> Best！
>> > > > > > > > > > > > > >> >> Xuyang
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
>> > > > > > > xingcanc@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >> >> >Hi Weiqing,
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early
>> > fire
>> > > > for
>> > > > > > time
>> > > > > > > > > > > interval
>> > > > > > > > > > > > > outer
>> > > > > > > > > > > > > >> >> >joins is feasible overall. I have a few
>> > > questions.
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >1. The current output stream of a time
>> > interval
>> > > > > outer
>> > > > > > > join
>> > > > > > > > > is
>> > > > > > > > > > an
>> > > > > > > > > > > > > >> >> >append-only stream. This change will make
>> it a
>> > > > > > potential
>> > > > > > > > > > > > retractable
>> > > > > > > > > > > > > >> >> >stream. I'm not sure if the planner
>> supports a
>> > > > > dynamic
>> > > > > > > > > output
>> > > > > > > > > > > type
>> > > > > > > > > > > > > like
>> > > > > > > > > > > > > >> >> >that. Could you add this part to your
>> design
>> > > doc?
>> > > > > > > > > > > > > >> >> >2. What's the use case when the downstream
>> > > > > components
>> > > > > > > need
>> > > > > > > > > to
>> > > > > > > > > > > get
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > >> >> early
>> > > > > > > > > > > > > >> >> >fired results regularly?
>> > > > > > > > > > > > > >> >> >3. The time interval join operator itself
>> is
>> > not
>> > > > > quite
>> > > > > > > > > > efficient
>> > > > > > > > > > > > > when
>> > > > > > > > > > > > > >> the
>> > > > > > > > > > > > > >> >> >state becomes large. Will there be any
>> extra
>> > > > > overhead
>> > > > > > > > after
>> > > > > > > > > > > > > introducing
>> > > > > > > > > > > > > >> >> >this feature?
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >Thanks,
>> > > > > > > > > > > > > >> >> >Xingcan
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing
>> Yang <
>> > > > > > > > > > > > > yangweiqing001@gmail.com>
>> > > > > > > > > > > > > >> >> >wrote:
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >> Hi all,
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the
>> > proposal
>> > > I
>> > > > > > shared
>> > > > > > > > on
>> > > > > > > > > > > early
>> > > > > > > > > > > > > fire
>> > > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
>> > > > greatly
>> > > > > > > > > appreciate
>> > > > > > > > > > > > your
>> > > > > > > > > > > > > >> >> feedback
>> > > > > > > > > > > > > >> >> >> or suggestions.
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> Here’s the link to the proposal document:
>> > Link
>> > > > > > > > > > > > > >> >> >> <
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> Thank you!
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> Best,
>> > > > > > > > > > > > > >> >> >> Weiqing
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing
>> > Yang
>> > > <
>> > > > > > > > > > > > > >> yangweiqing001@gmail.com
>> > > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > > >> >> >> wrote:
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >> >> > Hi all,
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
>> > > > > > introducing
>> > > > > > > > > early
>> > > > > > > > > > > fire
>> > > > > > > > > > > > > >> support
>> > > > > > > > > > > > > >> >> >> for
>> > > > > > > > > > > > > >> >> >> > Flink SQL interval joins.
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> > *Motivation*
>> > > > > > > > > > > > > >> >> >> > In many streaming applications,
>> > particularly
>> > > > > > > real-time
>> > > > > > > > > > > > analytics
>> > > > > > > > > > > > > >> and
>> > > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to
>> > obtain
>> > > > > > partial
>> > > > > > > > > > results
>> > > > > > > > > > > > > >> earlier
>> > > > > > > > > > > > > >> >> >> rather
>> > > > > > > > > > > > > >> >> >> > than waiting for full join conditions
>> to
>> > > > > finalize.
>> > > > > > > For
>> > > > > > > > > > Flink
>> > > > > > > > > > > > SQL
>> > > > > > > > > > > > > >> >> interval
>> > > > > > > > > > > > > >> >> >> > joins, results are typically delayed
>> until
>> > > > > > > watermarks
>> > > > > > > > > > ensure
>> > > > > > > > > > > > no
>> > > > > > > > > > > > > >> more
>> > > > > > > > > > > > > >> >> >> > matches can occur. This delay can be
>> > > > challenging
>> > > > > > for
>> > > > > > > > > > > scenarios
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > >> >> >> require
>> > > > > > > > > > > > > >> >> >> > fast feedback. Early fire support
>> > addresses
>> > > > this
>> > > > > > by
>> > > > > > > > > > emitting
>> > > > > > > > > > > > > >> >> intermediate
>> > > > > > > > > > > > > >> >> >> > results speculatively and using
>> > retractions
>> > > or
>> > > > > > > updates
>> > > > > > > > > to
>> > > > > > > > > > > > > maintain
>> > > > > > > > > > > > > >> >> >> eventual
>> > > > > > > > > > > > > >> >> >> > consistency and ensure correctness.
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
>> > > > > > > > > > > > > >> >> >> > <
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
>> > > refine
>> > > > > this
>> > > > > > > > > > feature.
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >> > Thanks,
>> > > > > > > > > > > > > >> >> >> > Weiqing
>> > > > > > > > > > > > > >> >> >> >
>> > > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > > >> >>
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Becket,
Thanks for the feedback! I’ve updated the design doc with a new
“Granularity of the Early Fire Hint” section: link
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.gle8q83bjlj5>
.
Here’s a quick summary:
- EARLY_FIRE is scoped to a single operator. If multiple operators in
one query can support early firing, Flink applies the hint to the first
matching operator and logs a warning to highlight the ambiguity. Subsequent
operators will not receive the hint.
- Split queries for different early-fire settings. If you need separate
early-fire configurations - or just want to avoid ambiguity - splitting the
SQL into multiple statements or views ensures each operator has its own
distinct EARLY_FIRE hint.
Please let me know if you have any other questions or suggestions.
Best,
Weiqing
On Fri, Feb 28, 2025 at 5:01 PM Becket Qin <be...@gmail.com> wrote:
> Thanks for updating the FLIP, Weiqing.
>
> Another question, what is the granularity of the early fire hint? Will the
> hint be applied to all the outer joins in the same block? What if in the
> query block there are also aggregations? For example:
>
> SELECT window_start, window_end, buyer, count(*) /*+
> EARLY_FIRE('delay'='5s'
> , 'time_mode'='rowtime|proctime') */
> FROM Orders o LEFT JOIN Shipments s
> WHERE o.id = s.order_id
> AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
> GROUP BY o.window_start, o.window_end, buyer;
>
> What will be the behavior? Will both the aggregation and interval join? Or
> will it only be applied to the join?
> It would be useful to define the behavior here.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Thanks for the feedback, Becket. I’ve heard similar concerns from other
> > reviewers, and I now agree that the `interval` option is likely not
> widely
> > used. In the FLIP, it was intended specifically for Interval Join, not
> as a
> > general early-fire mechanism. We can consider making it more generic in a
> > future update. Since it’s optional, I’ll go ahead and remove it for
> > clarity.
> >
> > Appreciate your input!
> >
> > Best,
> > Weiqing
> >
> > On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <be...@gmail.com> wrote:
> >
> > > Sorry for the late reply. I have a question regarding the configuration
> > of
> > > interval. I am wondering in which scenario the interval config will
> > > actually be used. Is the interval configuration only useful for
> windowed
> > > aggregation?
> > >
> > > For the join operations, early fire seems only applicable to the outer
> > join
> > > before there is a match. Once there is a match, the output of the join
> > > operation should be just event driven. For example, consider the
> > following
> > > scenario:
> > > 1. left outer join, and there is no match from the right side before
> the
> > > initial delay
> > > 2. a record [left, null] was emitted due to early fire
> > > 3. one of the following two cases must happen
> > > a. there is still no match event from the right side before the
> > > specific interval passes. In this case, we are not supposed to emit
> > another
> > > [left, null].
> > > b. if there is a match, a record of [left, right] should be emitted
> > > immediately regardless of the interval. If this happens, the interval
> > will
> > > be ignored from this point on.
> > >
> > > That said, if the configuration proposed in this FLIP is intended not
> > only
> > > for join, but for general purpose early fire, then the interval config
> > > makes sense.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <yangweiqing001@gmail.com
> >
> > > wrote:
> > >
> > > > Thank you all for reviewing. As there are no objections, I’ll move
> > > forward
> > > > with a vote.
> > > >
> > > > Best regards,
> > > > Weiqing
> > > >
> > > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> > > > vsowrira@asu.edu>
> > > > wrote:
> > > >
> > > > > Same here, I don't have any more questions. Thanks!
> > > > >
> > > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Weiqing,
> > > > > >
> > > > > > I don't have any more questions. The doc looks good to me.
> > > > > >
> > > > > > Thanks,
> > > > > > Xingcan
> > > > > >
> > > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > > > > vsowrira@asu.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Weiqing,
> > > > > > >
> > > > > > > Thanks, that makes sense! Looks like I missed it.
> > > > > > >
> > > > > > > Regards
> > > > > > > Venkata krishnan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > > > > yangweiqing001@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Venkata,
> > > > > > > >
> > > > > > > > * > where only one earlyFire is fired The DELAY *
> > > > > > > >
> > > > > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > > > > >
> > > > > > > > of the proposal can produce a single early fire per interval,
> > > > > aligning
> > > > > > > with
> > > > > > > > your suggestion. How it integrates with existing early-fire
> > > > > > > configurations
> > > > > > > > are mentioned here
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > > > > >.
> > > > > > > > Let me know if you have any further questions or feedback!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Weiqing
> > > > > > > >
> > > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > > > > vsowrira@asu.edu> wrote:
> > > > > > > >
> > > > > > > > > Thanks for the response!
> > > > > > > > >
> > > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > > early-fire
> > > > > > > > outputs
> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > > once.
> > > > > > After
> > > > > > > > that
> > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > >
> > > > > > > > > One more question, would it make sense to support a
> mechanism
> > > > > through
> > > > > > > > > configuration where only one earlyFire is fired for an
> > interval
> > > > or
> > > > > > > > window?
> > > > > > > > > This would also simplify the state overhead for use cases
> > where
> > > > > only
> > > > > > > one
> > > > > > > > > early fire is required within a window or interval.
> Thoughts?
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Venkata krishnan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > > > > yangweiqing001@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Venkata,
> > > > > > > > > >
> > > > > > > > > > Thanks for your feedback!
> > > > > > > > > >
> > > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > > early-fire
> > > > > > > > outputs
> > > > > > > > > > will occur repeatedly at every earlyFireInterval, not
> just
> > > > once.
> > > > > > > After
> > > > > > > > > that
> > > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > > >
> > > > > > > > > > We haven’t included a late-fire mechanism for interval
> > joins
> > > in
> > > > > > this
> > > > > > > > > FLIP,
> > > > > > > > > > but it’s certainly something we can consider for future
> > > > efforts!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Weiqing
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan
> Sowrirajan
> > <
> > > > > > > > > > vsowrira@asu.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Weiqing,
> > > > > > > > > > >
> > > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also
> > instead
> > > > of
> > > > > > > > > > controlling
> > > > > > > > > > > and configuring through Flink configs unlike the older
> > > window
> > > > > > > > > > aggregation,
> > > > > > > > > > > hints seems to be a much better approach. This enables
> a
> > > > > > > customizable
> > > > > > > > > > early
> > > > > > > > > > > fire behavior for individual interval joins.
> > > > > > > > > > >
> > > > > > > > > > > Couple of questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. Does the *early fire* emit an output every
> > > > earlyFireInterval
> > > > > > > time
> > > > > > > > or
> > > > > > > > > > > will it be a one time output emission and another
> output
> > > > > emitted
> > > > > > at
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > of the interval?
> > > > > > > > > > > 2. Are there plans to support *late fire *similar to
> the
> > > > window
> > > > > > > > > > > aggregations in later FLIPs?
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Venkata krishnan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > > > > yangweiqing001@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > > > > >
> > > > > > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any
> > > concerns?
> > > > > > > > > > > >
> > > > > > > > > > > > If no further objections arise from anyone, I’ll
> > proceed
> > > to
> > > > > > mark
> > > > > > > > FLIP
> > > > > > > > > > as
> > > > > > > > > > > > ready for voting.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Weiqing
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
> > > xyzhong131@163.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > LGTM overall. Thanks for updating. I have no
> problem
> > > and
> > > > +1
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > > > feature.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best！
> > > > > > > > > > > > > Xuyang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > > > > yangweiqing001@gmail.com
> > > > > > > >
> > > > > > > > > 写道：
> > > > > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated
> > the
> > > > > > > proposal
> > > > > > > > > doc
> > > > > > > > > > > > > >accordingly. Please feel free to take another look
> > and
> > > > let
> > > > > > me
> > > > > > > > know
> > > > > > > > > > if
> > > > > > > > > > > > you
> > > > > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >Best regards,
> > > > > > > > > > > > > >Weiqing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > > > > xyzhong131@163.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> After reading the new FLIP, I have no issues
> with
> > > the
> > > > > part
> > > > > > > > > `public
> > > > > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > > > > IntervalJoin
> > > > > > > > > node,
> > > > > > > > > > > > IIUC:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind
> of
> > > the
> > > > > > > > > IntervalJoin
> > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely
> on
> > > the
> > > > > > > > > > requirements
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - When early firing is disabled, the UpdateKind
> of
> > > the
> > > > > > > > > > IntervalJoin
> > > > > > > > > > > is
> > > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - Nevertheless, whether early firing is enabled
> or
> > > > > > disabled,
> > > > > > > > the
> > > > > > > > > > > > > >> IntervalJoin should always require its input to
> > keep
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
> > > NONE.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Best！
> > > > > > > > > > > > > >> Xuyang
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Thanks so much for the feedback - it was very
> > > > helpful!
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 1. The current output stream of a time
> > interval
> > > > > outer
> > > > > > > join
> > > > > > > > > is
> > > > > > > > > > an
> > > > > > > > > > > > > >> >append-only stream. This change will make it a
> > > > > potential
> > > > > > > > > > > retractable
> > > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> > > > dynamic
> > > > > > > output
> > > > > > > > > > type
> > > > > > > > > > > > like
> > > > > > > > > > > > > >> >that. Could you add this part to your design
> > doc?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - Yes, enabling early firing on time
> interval
> > > > outer
> > > > > > > joins
> > > > > > > > > can
> > > > > > > > > > > emit
> > > > > > > > > > > > > >> > retractions when previously emitted rows are
> > > > updated
> > > > > > or
> > > > > > > > > > > > invalidated
> > > > > > > > > > > > > by
> > > > > > > > > > > > > >> > later matches. I’ve updated the proposal
> > > (Planner
> > > > > > > > Awareness
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > and Changes in
> > > FlinkChangelogModeInferenceProgram
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > > >> >)
> > > > > > > > > > > > > >> > to clarify that the stream might switch from
> > > > > > append-only
> > > > > > > > to
> > > > > > > > > a
> > > > > > > > > > > > > >> > retract/upsert stream. Let me know if
> anything
> > > is
> > > > > > > missing.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > > > > components
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - The new INTERVAL option (in addition to
> > DELAY)
> > > > > > allows
> > > > > > > > > > periodic
> > > > > > > > > > > > > >> updates
> > > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
> > > delay.
> > > > > This
> > > > > > > > > > captures
> > > > > > > > > > > > how
> > > > > > > > > > > > > >> results
> > > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > > > > “Repeatedly”
> > > > > > > > > > option.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 3. The time interval join operator itself is
> > not
> > > > > quite
> > > > > > > > > > efficient
> > > > > > > > > > > > > when
> > > > > > > > > > > > > >> >the state becomes large. Will there be any
> extra
> > > > > overhead
> > > > > > > > after
> > > > > > > > > > > > > >> introducing
> > > > > > > > > > > > > >> >this feature?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > > > > potentially
> > > > > > > > > > > emitting
> > > > > > > > > > > > > >> > partial matches multiple times with
> retraction
> > > > > > (avoiding
> > > > > > > > > > > duplicate
> > > > > > > > > > > > > >> outputs
> > > > > > > > > > > > > >> > though). However, if it’s disabled, there is
> > no
> > > > > > > additional
> > > > > > > > > > cost.
> > > > > > > > > > > > > Most
> > > > > > > > > > > > > >> users
> > > > > > > > > > > > > >> > find the performance trade-off acceptable
> for
> > > the
> > > > > > > > real-time
> > > > > > > > > > > > > insights it
> > > > > > > > > > > > > >> > provides.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 1. Currently, there are some configs related
> > to
> > > > > early
> > > > > > > > firing
> > > > > > > > > > > > > available
> > > > > > > > > > > > > >> >to users:
> `table.exec.emit.early-fire.en**abled`
> > > and
> > > > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > > > > >> >**lay`.
> > > > > > > > > > > > > >> >Although their documentation states that they
> are
> > > > only
> > > > > > > > > applicable
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >Window operator, it seems possible that they
> may
> > > also
> > > > > be
> > > > > > > > > relevant
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >context of this FLIP. Otherwise, having
> different
> > > > early
> > > > > > > > firing
> > > > > > > > > > > > > behaviors
> > > > > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to
> avoid
> > > > > > > confusion.
> > > > > > > > > I’ve
> > > > > > > > > > > > > added a
> > > > > > > > > > > > > >> > section
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >in
> > > > > > > > > > > > > >> > the proposal highlighting that we should
> align
> > > > > > > hint-based
> > > > > > > > > > > interval
> > > > > > > > > > > > > join
> > > > > > > > > > > > > >> > configurations with the existing
> > > table.exec.emit.*
> > > > > > > > settings.
> > > > > > > > > > > > > >> Suggestions
> > > > > > > > > > > > > >> > on how to make the unification are welcome!
> We
> > > > plan
> > > > > to
> > > > > > > > > extend
> > > > > > > > > > > > early
> > > > > > > > > > > > > >> firing
> > > > > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
> > > Similar
> > > > > to
> > > > > > > > point
> > > > > > > > > 1,
> > > > > > > > > > > > > perhaps
> > > > > > > > > > > > > >> >we can introduce it to other window-related
> > > operators
> > > > > in
> > > > > > > the
> > > > > > > > > > > future.>
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >> >You need to modify the
> > > > > FlinkChangelogModeInferenceProgram
> > > > > > > to
> > > > > > > > > > ensure
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> >downstream nodes of interval joins with early
> > > firing
> > > > > > > enabled
> > > > > > > > > are
> > > > > > > > > > > > aware
> > > > > > > > > > > > > of
> > > > > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - We agree that time_mode could be
> introduced
> > to
> > > > > other
> > > > > > > > > > > > window-based
> > > > > > > > > > > > > >> > operators down the road. We also want to
> > support
> > > > > early
> > > > > > > > fire
> > > > > > > > > > for
> > > > > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added
> > the
> > > > code
> > > > > > > > change
> > > > > > > > > on
> > > > > > > > > > > it
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > in the proposal.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
> > > updated
> > > > > the
> > > > > > > > > > proposal
> > > > > > > > > > > > with
> > > > > > > > > > > > > >> >these points. Please let me know if there’s
> > > anything
> > > > > > else I
> > > > > > > > > > should
> > > > > > > > > > > > > >> address.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Best,
> > > > > > > > > > > > > >> >Weiqing
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > > > > xyzhong131@163.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this
> FLIP.
> > I
> > > > > have a
> > > > > > > few
> > > > > > > > > > > > > questions:
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 1. Currently, there are some configs related
> to
> > > > early
> > > > > > > > firing
> > > > > > > > > > > > > available
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled`
> and
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
> > > their
> > > > > > > > > > documentation
> > > > > > > > > > > > > states
> > > > > > > > > > > > > >> >> that they are only applicable to the Window
> > > > operator,
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> it seems possible that they may also be
> > relevant
> > > in
> > > > > the
> > > > > > > > > context
> > > > > > > > > > > of
> > > > > > > > > > > > > this
> > > > > > > > > > > > > >> >> FLIP. Otherwise, having different early
> firing
> > > > > > behaviors
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
> > > Similar
> > > > to
> > > > > > > point
> > > > > > > > > 1,
> > > > > > > > > > > > > perhaps
> > > > > > > > > > > > > >> we
> > > > > > > > > > > > > >> >> can introduce it to other window-related
> > > operators
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> in the future.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 3. You need to modify the
> > > > > > > > FlinkChangelogModeInferenceProgram
> > > > > > > > > to
> > > > > > > > > > > > > ensure
> > > > > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> early firing enabled are aware of retract or
> > > upsert
> > > > > > > > messages.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> --
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> Best！
> > > > > > > > > > > > > >> >> Xuyang
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > > > > xingcanc@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early
> > fire
> > > > for
> > > > > > time
> > > > > > > > > > > interval
> > > > > > > > > > > > > outer
> > > > > > > > > > > > > >> >> >joins is feasible overall. I have a few
> > > questions.
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >1. The current output stream of a time
> > interval
> > > > > outer
> > > > > > > join
> > > > > > > > > is
> > > > > > > > > > an
> > > > > > > > > > > > > >> >> >append-only stream. This change will make
> it a
> > > > > > potential
> > > > > > > > > > > > retractable
> > > > > > > > > > > > > >> >> >stream. I'm not sure if the planner
> supports a
> > > > > dynamic
> > > > > > > > > output
> > > > > > > > > > > type
> > > > > > > > > > > > > like
> > > > > > > > > > > > > >> >> >that. Could you add this part to your design
> > > doc?
> > > > > > > > > > > > > >> >> >2. What's the use case when the downstream
> > > > > components
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >> early
> > > > > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > > > > >> >> >3. The time interval join operator itself is
> > not
> > > > > quite
> > > > > > > > > > efficient
> > > > > > > > > > > > > when
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > > > > overhead
> > > > > > > > after
> > > > > > > > > > > > > introducing
> > > > > > > > > > > > > >> >> >this feature?
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang
> <
> > > > > > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > > > >> >> >wrote:
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the
> > proposal
> > > I
> > > > > > shared
> > > > > > > > on
> > > > > > > > > > > early
> > > > > > > > > > > > > fire
> > > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> > > > greatly
> > > > > > > > > appreciate
> > > > > > > > > > > > your
> > > > > > > > > > > > > >> >> feedback
> > > > > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Here’s the link to the proposal document:
> > Link
> > > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Best,
> > > > > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing
> > Yang
> > > <
> > > > > > > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > > > > introducing
> > > > > > > > > early
> > > > > > > > > > > fire
> > > > > > > > > > > > > >> support
> > > > > > > > > > > > > >> >> >> for
> > > > > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > > > > >> >> >> > In many streaming applications,
> > particularly
> > > > > > > real-time
> > > > > > > > > > > > analytics
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to
> > obtain
> > > > > > partial
> > > > > > > > > > results
> > > > > > > > > > > > > >> earlier
> > > > > > > > > > > > > >> >> >> rather
> > > > > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > > > > finalize.
> > > > > > > For
> > > > > > > > > > Flink
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > >> >> interval
> > > > > > > > > > > > > >> >> >> > joins, results are typically delayed
> until
> > > > > > > watermarks
> > > > > > > > > > ensure
> > > > > > > > > > > > no
> > > > > > > > > > > > > >> more
> > > > > > > > > > > > > >> >> >> > matches can occur. This delay can be
> > > > challenging
> > > > > > for
> > > > > > > > > > > scenarios
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> >> >> require
> > > > > > > > > > > > > >> >> >> > fast feedback. Early fire support
> > addresses
> > > > this
> > > > > > by
> > > > > > > > > > emitting
> > > > > > > > > > > > > >> >> intermediate
> > > > > > > > > > > > > >> >> >> > results speculatively and using
> > retractions
> > > or
> > > > > > > updates
> > > > > > > > > to
> > > > > > > > > > > > > maintain
> > > > > > > > > > > > > >> >> >> eventual
> > > > > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > > > > >> >> >> > <
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
> > > refine
> > > > > this
> > > > > > > > > > feature.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Becket Qin <be...@gmail.com>.
Thanks for updating the FLIP, Weiqing.
Another question, what is the granularity of the early fire hint? Will the
hint be applied to all the outer joins in the same block? What if in the
query block there are also aggregations? For example:
SELECT window_start, window_end, buyer, count(*) /*+ EARLY_FIRE('delay'='5s'
, 'time_mode'='rowtime|proctime') */
FROM Orders o LEFT JOIN Shipments s
WHERE o.id = s.order_id
AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
GROUP BY o.window_start, o.window_end, buyer;
What will be the behavior? Will both the aggregation and interval join? Or
will it only be applied to the join?
It would be useful to define the behavior here.
Thanks,
Jiangjie (Becket) Qin
On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Thanks for the feedback, Becket. I’ve heard similar concerns from other
> reviewers, and I now agree that the `interval` option is likely not widely
> used. In the FLIP, it was intended specifically for Interval Join, not as a
> general early-fire mechanism. We can consider making it more generic in a
> future update. Since it’s optional, I’ll go ahead and remove it for
> clarity.
>
> Appreciate your input!
>
> Best,
> Weiqing
>
> On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <be...@gmail.com> wrote:
>
> > Sorry for the late reply. I have a question regarding the configuration
> of
> > interval. I am wondering in which scenario the interval config will
> > actually be used. Is the interval configuration only useful for windowed
> > aggregation?
> >
> > For the join operations, early fire seems only applicable to the outer
> join
> > before there is a match. Once there is a match, the output of the join
> > operation should be just event driven. For example, consider the
> following
> > scenario:
> > 1. left outer join, and there is no match from the right side before the
> > initial delay
> > 2. a record [left, null] was emitted due to early fire
> > 3. one of the following two cases must happen
> > a. there is still no match event from the right side before the
> > specific interval passes. In this case, we are not supposed to emit
> another
> > [left, null].
> > b. if there is a match, a record of [left, right] should be emitted
> > immediately regardless of the interval. If this happens, the interval
> will
> > be ignored from this point on.
> >
> > That said, if the configuration proposed in this FLIP is intended not
> only
> > for join, but for general purpose early fire, then the interval config
> > makes sense.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <ya...@gmail.com>
> > wrote:
> >
> > > Thank you all for reviewing. As there are no objections, I’ll move
> > forward
> > > with a vote.
> > >
> > > Best regards,
> > > Weiqing
> > >
> > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> > > vsowrira@asu.edu>
> > > wrote:
> > >
> > > > Same here, I don't have any more questions. Thanks!
> > > >
> > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com>
> wrote:
> > > >
> > > > > Hi Weiqing,
> > > > >
> > > > > I don't have any more questions. The doc looks good to me.
> > > > >
> > > > > Thanks,
> > > > > Xingcan
> > > > >
> > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > > > vsowrira@asu.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Weiqing,
> > > > > >
> > > > > > Thanks, that makes sense! Looks like I missed it.
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Venkata,
> > > > > > >
> > > > > > > * > where only one earlyFire is fired The DELAY *
> > > > > > >
> > > > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > > > >
> > > > > > > of the proposal can produce a single early fire per interval,
> > > > aligning
> > > > > > with
> > > > > > > your suggestion. How it integrates with existing early-fire
> > > > > > configurations
> > > > > > > are mentioned here
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > > > >.
> > > > > > > Let me know if you have any further questions or feedback!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Weiqing
> > > > > > >
> > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > > > vsowrira@asu.edu> wrote:
> > > > > > >
> > > > > > > > Thanks for the response!
> > > > > > > >
> > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > early-fire
> > > > > > > outputs
> > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > once.
> > > > > After
> > > > > > > that
> > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > >
> > > > > > > > One more question, would it make sense to support a mechanism
> > > > through
> > > > > > > > configuration where only one earlyFire is fired for an
> interval
> > > or
> > > > > > > window?
> > > > > > > > This would also simplify the state overhead for use cases
> where
> > > > only
> > > > > > one
> > > > > > > > early fire is required within a window or interval. Thoughts?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Venkata krishnan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > > > yangweiqing001@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Venkata,
> > > > > > > > >
> > > > > > > > > Thanks for your feedback!
> > > > > > > > >
> > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > early-fire
> > > > > > > outputs
> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > > once.
> > > > > > After
> > > > > > > > that
> > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > >
> > > > > > > > > We haven’t included a late-fire mechanism for interval
> joins
> > in
> > > > > this
> > > > > > > > FLIP,
> > > > > > > > > but it’s certainly something we can consider for future
> > > efforts!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Weiqing
> > > > > > > > >
> > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan
> <
> > > > > > > > > vsowrira@asu.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Weiqing,
> > > > > > > > > >
> > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also
> instead
> > > of
> > > > > > > > > controlling
> > > > > > > > > > and configuring through Flink configs unlike the older
> > window
> > > > > > > > > aggregation,
> > > > > > > > > > hints seems to be a much better approach. This enables a
> > > > > > customizable
> > > > > > > > > early
> > > > > > > > > > fire behavior for individual interval joins.
> > > > > > > > > >
> > > > > > > > > > Couple of questions:
> > > > > > > > > >
> > > > > > > > > > 1. Does the *early fire* emit an output every
> > > earlyFireInterval
> > > > > > time
> > > > > > > or
> > > > > > > > > > will it be a one time output emission and another output
> > > > emitted
> > > > > at
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > of the interval?
> > > > > > > > > > 2. Are there plans to support *late fire *similar to the
> > > window
> > > > > > > > > > aggregations in later FLIPs?
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Venkata krishnan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > > > yangweiqing001@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > > > >
> > > > > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any
> > concerns?
> > > > > > > > > > >
> > > > > > > > > > > If no further objections arise from anyone, I’ll
> proceed
> > to
> > > > > mark
> > > > > > > FLIP
> > > > > > > > > as
> > > > > > > > > > > ready for voting.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Weiqing
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
> > xyzhong131@163.com
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > LGTM overall. Thanks for updating. I have no problem
> > and
> > > +1
> > > > > for
> > > > > > > > this
> > > > > > > > > > > > feature.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Best！
> > > > > > > > > > > > Xuyang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > > > yangweiqing001@gmail.com
> > > > > > >
> > > > > > > > 写道：
> > > > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > > > >
> > > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated
> the
> > > > > > proposal
> > > > > > > > doc
> > > > > > > > > > > > >accordingly. Please feel free to take another look
> and
> > > let
> > > > > me
> > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > you
> > > > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > > > >
> > > > > > > > > > > > >Best regards,
> > > > > > > > > > > > >Weiqing
> > > > > > > > > > > > >
> > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > > > xyzhong131@163.com>
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> After reading the new FLIP, I have no issues with
> > the
> > > > part
> > > > > > > > `public
> > > > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > > > IntervalJoin
> > > > > > > > node,
> > > > > > > > > > > IIUC:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind of
> > the
> > > > > > > > IntervalJoin
> > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on
> > the
> > > > > > > > > requirements
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - When early firing is disabled, the UpdateKind of
> > the
> > > > > > > > > IntervalJoin
> > > > > > > > > > is
> > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > > > > disabled,
> > > > > > > the
> > > > > > > > > > > > >> IntervalJoin should always require its input to
> keep
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
> > NONE.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Best！
> > > > > > > > > > > > >> Xuyang
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Thanks so much for the feedback - it was very
> > > helpful!
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 1. The current output stream of a time
> interval
> > > > outer
> > > > > > join
> > > > > > > > is
> > > > > > > > > an
> > > > > > > > > > > > >> >append-only stream. This change will make it a
> > > > potential
> > > > > > > > > > retractable
> > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> > > dynamic
> > > > > > output
> > > > > > > > > type
> > > > > > > > > > > like
> > > > > > > > > > > > >> >that. Could you add this part to your design
> doc?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - Yes, enabling early firing on time interval
> > > outer
> > > > > > joins
> > > > > > > > can
> > > > > > > > > > emit
> > > > > > > > > > > > >> > retractions when previously emitted rows are
> > > updated
> > > > > or
> > > > > > > > > > > invalidated
> > > > > > > > > > > > by
> > > > > > > > > > > > >> > later matches. I’ve updated the proposal
> > (Planner
> > > > > > > Awareness
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > and Changes in
> > FlinkChangelogModeInferenceProgram
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > >> >)
> > > > > > > > > > > > >> > to clarify that the stream might switch from
> > > > > append-only
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > > > >> > retract/upsert stream. Let me know if anything
> > is
> > > > > > missing.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > > > components
> > > > > > need
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - The new INTERVAL option (in addition to
> DELAY)
> > > > > allows
> > > > > > > > > periodic
> > > > > > > > > > > > >> updates
> > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
> > delay.
> > > > This
> > > > > > > > > captures
> > > > > > > > > > > how
> > > > > > > > > > > > >> results
> > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > > > “Repeatedly”
> > > > > > > > > option.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 3. The time interval join operator itself is
> not
> > > > quite
> > > > > > > > > efficient
> > > > > > > > > > > > when
> > > > > > > > > > > > >> >the state becomes large. Will there be any extra
> > > > overhead
> > > > > > > after
> > > > > > > > > > > > >> introducing
> > > > > > > > > > > > >> >this feature?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > > > potentially
> > > > > > > > > > emitting
> > > > > > > > > > > > >> > partial matches multiple times with retraction
> > > > > (avoiding
> > > > > > > > > > duplicate
> > > > > > > > > > > > >> outputs
> > > > > > > > > > > > >> > though). However, if it’s disabled, there is
> no
> > > > > > additional
> > > > > > > > > cost.
> > > > > > > > > > > > Most
> > > > > > > > > > > > >> users
> > > > > > > > > > > > >> > find the performance trade-off acceptable for
> > the
> > > > > > > real-time
> > > > > > > > > > > > insights it
> > > > > > > > > > > > >> > provides.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 1. Currently, there are some configs related
> to
> > > > early
> > > > > > > firing
> > > > > > > > > > > > available
> > > > > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled`
> > and
> > > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > > > >> >**lay`.
> > > > > > > > > > > > >> >Although their documentation states that they are
> > > only
> > > > > > > > applicable
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >Window operator, it seems possible that they may
> > also
> > > > be
> > > > > > > > relevant
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >context of this FLIP. Otherwise, having different
> > > early
> > > > > > > firing
> > > > > > > > > > > > behaviors
> > > > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > > > > confusion.
> > > > > > > > I’ve
> > > > > > > > > > > > added a
> > > > > > > > > > > > >> > section
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >in
> > > > > > > > > > > > >> > the proposal highlighting that we should align
> > > > > > hint-based
> > > > > > > > > > interval
> > > > > > > > > > > > join
> > > > > > > > > > > > >> > configurations with the existing
> > table.exec.emit.*
> > > > > > > settings.
> > > > > > > > > > > > >> Suggestions
> > > > > > > > > > > > >> > on how to make the unification are welcome! We
> > > plan
> > > > to
> > > > > > > > extend
> > > > > > > > > > > early
> > > > > > > > > > > > >> firing
> > > > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
> > Similar
> > > > to
> > > > > > > point
> > > > > > > > 1,
> > > > > > > > > > > > perhaps
> > > > > > > > > > > > >> >we can introduce it to other window-related
> > operators
> > > > in
> > > > > > the
> > > > > > > > > > future.>
> > > > > > > > > > > > 3.
> > > > > > > > > > > > >> >You need to modify the
> > > > FlinkChangelogModeInferenceProgram
> > > > > > to
> > > > > > > > > ensure
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >downstream nodes of interval joins with early
> > firing
> > > > > > enabled
> > > > > > > > are
> > > > > > > > > > > aware
> > > > > > > > > > > > of
> > > > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - We agree that time_mode could be introduced
> to
> > > > other
> > > > > > > > > > > window-based
> > > > > > > > > > > > >> > operators down the road. We also want to
> support
> > > > early
> > > > > > > fire
> > > > > > > > > for
> > > > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added
> the
> > > code
> > > > > > > change
> > > > > > > > on
> > > > > > > > > > it
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > in the proposal.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
> > updated
> > > > the
> > > > > > > > > proposal
> > > > > > > > > > > with
> > > > > > > > > > > > >> >these points. Please let me know if there’s
> > anything
> > > > > else I
> > > > > > > > > should
> > > > > > > > > > > > >> address.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Best,
> > > > > > > > > > > > >> >Weiqing
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > > > xyzhong131@163.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP.
> I
> > > > have a
> > > > > > few
> > > > > > > > > > > > questions:
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 1. Currently, there are some configs related to
> > > early
> > > > > > > firing
> > > > > > > > > > > > available
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
> > their
> > > > > > > > > documentation
> > > > > > > > > > > > states
> > > > > > > > > > > > >> >> that they are only applicable to the Window
> > > operator,
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> it seems possible that they may also be
> relevant
> > in
> > > > the
> > > > > > > > context
> > > > > > > > > > of
> > > > > > > > > > > > this
> > > > > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > > > > behaviors
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
> > Similar
> > > to
> > > > > > point
> > > > > > > > 1,
> > > > > > > > > > > > perhaps
> > > > > > > > > > > > >> we
> > > > > > > > > > > > >> >> can introduce it to other window-related
> > operators
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> in the future.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 3. You need to modify the
> > > > > > > FlinkChangelogModeInferenceProgram
> > > > > > > > to
> > > > > > > > > > > > ensure
> > > > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> early firing enabled are aware of retract or
> > upsert
> > > > > > > messages.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> --
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> Best！
> > > > > > > > > > > > >> >> Xuyang
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > > > xingcanc@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early
> fire
> > > for
> > > > > time
> > > > > > > > > > interval
> > > > > > > > > > > > outer
> > > > > > > > > > > > >> >> >joins is feasible overall. I have a few
> > questions.
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >1. The current output stream of a time
> interval
> > > > outer
> > > > > > join
> > > > > > > > is
> > > > > > > > > an
> > > > > > > > > > > > >> >> >append-only stream. This change will make it a
> > > > > potential
> > > > > > > > > > > retractable
> > > > > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> > > > dynamic
> > > > > > > > output
> > > > > > > > > > type
> > > > > > > > > > > > like
> > > > > > > > > > > > >> >> >that. Could you add this part to your design
> > doc?
> > > > > > > > > > > > >> >> >2. What's the use case when the downstream
> > > > components
> > > > > > need
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >> early
> > > > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > > > >> >> >3. The time interval join operator itself is
> not
> > > > quite
> > > > > > > > > efficient
> > > > > > > > > > > > when
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > > > overhead
> > > > > > > after
> > > > > > > > > > > > introducing
> > > > > > > > > > > > >> >> >this feature?
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > > >> >> >wrote:
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the
> proposal
> > I
> > > > > shared
> > > > > > > on
> > > > > > > > > > early
> > > > > > > > > > > > fire
> > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> > > greatly
> > > > > > > > appreciate
> > > > > > > > > > > your
> > > > > > > > > > > > >> >> feedback
> > > > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Here’s the link to the proposal document:
> Link
> > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Best,
> > > > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing
> Yang
> > <
> > > > > > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > > > introducing
> > > > > > > > early
> > > > > > > > > > fire
> > > > > > > > > > > > >> support
> > > > > > > > > > > > >> >> >> for
> > > > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > > > >> >> >> > In many streaming applications,
> particularly
> > > > > > real-time
> > > > > > > > > > > analytics
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to
> obtain
> > > > > partial
> > > > > > > > > results
> > > > > > > > > > > > >> earlier
> > > > > > > > > > > > >> >> >> rather
> > > > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > > > finalize.
> > > > > > For
> > > > > > > > > Flink
> > > > > > > > > > > SQL
> > > > > > > > > > > > >> >> interval
> > > > > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > > > > watermarks
> > > > > > > > > ensure
> > > > > > > > > > > no
> > > > > > > > > > > > >> more
> > > > > > > > > > > > >> >> >> > matches can occur. This delay can be
> > > challenging
> > > > > for
> > > > > > > > > > scenarios
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >> >> require
> > > > > > > > > > > > >> >> >> > fast feedback. Early fire support
> addresses
> > > this
> > > > > by
> > > > > > > > > emitting
> > > > > > > > > > > > >> >> intermediate
> > > > > > > > > > > > >> >> >> > results speculatively and using
> retractions
> > or
> > > > > > updates
> > > > > > > > to
> > > > > > > > > > > > maintain
> > > > > > > > > > > > >> >> >> eventual
> > > > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > > > >> >> >> > <
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
> > refine
> > > > this
> > > > > > > > > feature.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Thanks for the feedback, Becket. I’ve heard similar concerns from other
reviewers, and I now agree that the `interval` option is likely not widely
used. In the FLIP, it was intended specifically for Interval Join, not as a
general early-fire mechanism. We can consider making it more generic in a
future update. Since it’s optional, I’ll go ahead and remove it for
clarity.
Appreciate your input!
Best,
Weiqing
On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <be...@gmail.com> wrote:
> Sorry for the late reply. I have a question regarding the configuration of
> interval. I am wondering in which scenario the interval config will
> actually be used. Is the interval configuration only useful for windowed
> aggregation?
>
> For the join operations, early fire seems only applicable to the outer join
> before there is a match. Once there is a match, the output of the join
> operation should be just event driven. For example, consider the following
> scenario:
> 1. left outer join, and there is no match from the right side before the
> initial delay
> 2. a record [left, null] was emitted due to early fire
> 3. one of the following two cases must happen
> a. there is still no match event from the right side before the
> specific interval passes. In this case, we are not supposed to emit another
> [left, null].
> b. if there is a match, a record of [left, right] should be emitted
> immediately regardless of the interval. If this happens, the interval will
> be ignored from this point on.
>
> That said, if the configuration proposed in this FLIP is intended not only
> for join, but for general purpose early fire, then the interval config
> makes sense.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Thank you all for reviewing. As there are no objections, I’ll move
> forward
> > with a vote.
> >
> > Best regards,
> > Weiqing
> >
> > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> > vsowrira@asu.edu>
> > wrote:
> >
> > > Same here, I don't have any more questions. Thanks!
> > >
> > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com> wrote:
> > >
> > > > Hi Weiqing,
> > > >
> > > > I don't have any more questions. The doc looks good to me.
> > > >
> > > > Thanks,
> > > > Xingcan
> > > >
> > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > > vsowrira@asu.edu>
> > > > wrote:
> > > >
> > > > > Hi Weiqing,
> > > > >
> > > > > Thanks, that makes sense! Looks like I missed it.
> > > > >
> > > > > Regards
> > > > > Venkata krishnan
> > > > >
> > > > >
> > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > > yangweiqing001@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Venkata,
> > > > > >
> > > > > > * > where only one earlyFire is fired The DELAY *
> > > > > >
> > > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > > >
> > > > > > of the proposal can produce a single early fire per interval,
> > > aligning
> > > > > with
> > > > > > your suggestion. How it integrates with existing early-fire
> > > > > configurations
> > > > > > are mentioned here
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > > >.
> > > > > > Let me know if you have any further questions or feedback!
> > > > > >
> > > > > > Thanks,
> > > > > > Weiqing
> > > > > >
> > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > > vsowrira@asu.edu> wrote:
> > > > > >
> > > > > > > Thanks for the response!
> > > > > > >
> > > > > > > > If the optional `interval` in the proposal is enabled,
> > early-fire
> > > > > > outputs
> > > > > > > will occur repeatedly at every earlyFireInterval, not just
> once.
> > > > After
> > > > > > that
> > > > > > > interval concludes, there will also be a final emission.
> > > > > > >
> > > > > > > One more question, would it make sense to support a mechanism
> > > through
> > > > > > > configuration where only one earlyFire is fired for an interval
> > or
> > > > > > window?
> > > > > > > This would also simplify the state overhead for use cases where
> > > only
> > > > > one
> > > > > > > early fire is required within a window or interval. Thoughts?
> > > > > > >
> > > > > > > Regards
> > > > > > > Venkata krishnan
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Venkata,
> > > > > > > >
> > > > > > > > Thanks for your feedback!
> > > > > > > >
> > > > > > > > If the optional `interval` in the proposal is enabled,
> > early-fire
> > > > > > outputs
> > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > once.
> > > > > After
> > > > > > > that
> > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > >
> > > > > > > > We haven’t included a late-fire mechanism for interval joins
> in
> > > > this
> > > > > > > FLIP,
> > > > > > > > but it’s certainly something we can consider for future
> > efforts!
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Weiqing
> > > > > > > >
> > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > > > > > vsowrira@asu.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Weiqing,
> > > > > > > > >
> > > > > > > > > Thanks, this is a great addition to Flink SQL. Also instead
> > of
> > > > > > > > controlling
> > > > > > > > > and configuring through Flink configs unlike the older
> window
> > > > > > > > aggregation,
> > > > > > > > > hints seems to be a much better approach. This enables a
> > > > > customizable
> > > > > > > > early
> > > > > > > > > fire behavior for individual interval joins.
> > > > > > > > >
> > > > > > > > > Couple of questions:
> > > > > > > > >
> > > > > > > > > 1. Does the *early fire* emit an output every
> > earlyFireInterval
> > > > > time
> > > > > > or
> > > > > > > > > will it be a one time output emission and another output
> > > emitted
> > > > at
> > > > > > the
> > > > > > > > end
> > > > > > > > > of the interval?
> > > > > > > > > 2. Are there plans to support *late fire *similar to the
> > window
> > > > > > > > > aggregations in later FLIPs?
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Venkata krishnan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > > yangweiqing001@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > > >
> > > > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any
> concerns?
> > > > > > > > > >
> > > > > > > > > > If no further objections arise from anyone, I’ll proceed
> to
> > > > mark
> > > > > > FLIP
> > > > > > > > as
> > > > > > > > > > ready for voting.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Weiqing
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
> xyzhong131@163.com
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > LGTM overall. Thanks for updating. I have no problem
> and
> > +1
> > > > for
> > > > > > > this
> > > > > > > > > > > feature.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Best！
> > > > > > > > > > > Xuyang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > > yangweiqing001@gmail.com
> > > > > >
> > > > > > > 写道：
> > > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > > >
> > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated the
> > > > > proposal
> > > > > > > doc
> > > > > > > > > > > >accordingly. Please feel free to take another look and
> > let
> > > > me
> > > > > > know
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > > >
> > > > > > > > > > > >Best regards,
> > > > > > > > > > > >Weiqing
> > > > > > > > > > > >
> > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > > xyzhong131@163.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > > >>
> > > > > > > > > > > >> After reading the new FLIP, I have no issues with
> the
> > > part
> > > > > > > `public
> > > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > > >>
> > > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > > IntervalJoin
> > > > > > > node,
> > > > > > > > > > IIUC:
> > > > > > > > > > > >>
> > > > > > > > > > > >> - When early firing is enabled, the UpdateKind of
> the
> > > > > > > IntervalJoin
> > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > > >>
> > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on
> the
> > > > > > > > requirements
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > > >>
> > > > > > > > > > > >> - When early firing is disabled, the UpdateKind of
> the
> > > > > > > > IntervalJoin
> > > > > > > > > is
> > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > > >>
> > > > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > > > disabled,
> > > > > > the
> > > > > > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > > > > > >>
> > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
> NONE.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >>
> > > > > > > > > > > >> Best！
> > > > > > > > > > > >> Xuyang
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >Thanks so much for the feedback - it was very
> > helpful!
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >*> 1. The current output stream of a time interval
> > > outer
> > > > > join
> > > > > > > is
> > > > > > > > an
> > > > > > > > > > > >> >append-only stream. This change will make it a
> > > potential
> > > > > > > > > retractable
> > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> > dynamic
> > > > > output
> > > > > > > > type
> > > > > > > > > > like
> > > > > > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - Yes, enabling early firing on time interval
> > outer
> > > > > joins
> > > > > > > can
> > > > > > > > > emit
> > > > > > > > > > > >> > retractions when previously emitted rows are
> > updated
> > > > or
> > > > > > > > > > invalidated
> > > > > > > > > > > by
> > > > > > > > > > > >> > later matches. I’ve updated the proposal
> (Planner
> > > > > > Awareness
> > > > > > > > > > > >> > <
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > and Changes in
> FlinkChangelogModeInferenceProgram
> > > > > > > > > > > >> > <
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > >> >)
> > > > > > > > > > > >> > to clarify that the stream might switch from
> > > > append-only
> > > > > > to
> > > > > > > a
> > > > > > > > > > > >> > retract/upsert stream. Let me know if anything
> is
> > > > > missing.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > > components
> > > > > need
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > the
> > > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - The new INTERVAL option (in addition to DELAY)
> > > > allows
> > > > > > > > periodic
> > > > > > > > > > > >> updates
> > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
> delay.
> > > This
> > > > > > > > captures
> > > > > > > > > > how
> > > > > > > > > > > >> results
> > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > > “Repeatedly”
> > > > > > > > option.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >*> 3. The time interval join operator itself is not
> > > quite
> > > > > > > > efficient
> > > > > > > > > > > when
> > > > > > > > > > > >> >the state becomes large. Will there be any extra
> > > overhead
> > > > > > after
> > > > > > > > > > > >> introducing
> > > > > > > > > > > >> >this feature?*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > > potentially
> > > > > > > > > emitting
> > > > > > > > > > > >> > partial matches multiple times with retraction
> > > > (avoiding
> > > > > > > > > duplicate
> > > > > > > > > > > >> outputs
> > > > > > > > > > > >> > though). However, if it’s disabled, there is no
> > > > > additional
> > > > > > > > cost.
> > > > > > > > > > > Most
> > > > > > > > > > > >> users
> > > > > > > > > > > >> > find the performance trade-off acceptable for
> the
> > > > > > real-time
> > > > > > > > > > > insights it
> > > > > > > > > > > >> > provides.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >*> 1. Currently, there are some configs related to
> > > early
> > > > > > firing
> > > > > > > > > > > available
> > > > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled`
> and
> > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > > >> >**lay`.
> > > > > > > > > > > >> >Although their documentation states that they are
> > only
> > > > > > > applicable
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > >> >Window operator, it seems possible that they may
> also
> > > be
> > > > > > > relevant
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > >> >context of this FLIP. Otherwise, having different
> > early
> > > > > > firing
> > > > > > > > > > > behaviors
> > > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > > > confusion.
> > > > > > > I’ve
> > > > > > > > > > > added a
> > > > > > > > > > > >> > section
> > > > > > > > > > > >> > <
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >in
> > > > > > > > > > > >> > the proposal highlighting that we should align
> > > > > hint-based
> > > > > > > > > interval
> > > > > > > > > > > join
> > > > > > > > > > > >> > configurations with the existing
> table.exec.emit.*
> > > > > > settings.
> > > > > > > > > > > >> Suggestions
> > > > > > > > > > > >> > on how to make the unification are welcome! We
> > plan
> > > to
> > > > > > > extend
> > > > > > > > > > early
> > > > > > > > > > > >> firing
> > > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
> Similar
> > > to
> > > > > > point
> > > > > > > 1,
> > > > > > > > > > > perhaps
> > > > > > > > > > > >> >we can introduce it to other window-related
> operators
> > > in
> > > > > the
> > > > > > > > > future.>
> > > > > > > > > > > 3.
> > > > > > > > > > > >> >You need to modify the
> > > FlinkChangelogModeInferenceProgram
> > > > > to
> > > > > > > > ensure
> > > > > > > > > > > that
> > > > > > > > > > > >> >downstream nodes of interval joins with early
> firing
> > > > > enabled
> > > > > > > are
> > > > > > > > > > aware
> > > > > > > > > > > of
> > > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > - We agree that time_mode could be introduced to
> > > other
> > > > > > > > > > window-based
> > > > > > > > > > > >> > operators down the road. We also want to support
> > > early
> > > > > > fire
> > > > > > > > for
> > > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the
> > code
> > > > > > change
> > > > > > > on
> > > > > > > > > it
> > > > > > > > > > > >> > <
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > in the proposal.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
> updated
> > > the
> > > > > > > > proposal
> > > > > > > > > > with
> > > > > > > > > > > >> >these points. Please let me know if there’s
> anything
> > > > else I
> > > > > > > > should
> > > > > > > > > > > >> address.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >Best,
> > > > > > > > > > > >> >Weiqing
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > > xyzhong131@163.com>
> > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I
> > > have a
> > > > > few
> > > > > > > > > > > questions:
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> 1. Currently, there are some configs related to
> > early
> > > > > > firing
> > > > > > > > > > > available
> > > > > > > > > > > >> to
> > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
> their
> > > > > > > > documentation
> > > > > > > > > > > states
> > > > > > > > > > > >> >> that they are only applicable to the Window
> > operator,
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> it seems possible that they may also be relevant
> in
> > > the
> > > > > > > context
> > > > > > > > > of
> > > > > > > > > > > this
> > > > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > > > behaviors
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
> Similar
> > to
> > > > > point
> > > > > > > 1,
> > > > > > > > > > > perhaps
> > > > > > > > > > > >> we
> > > > > > > > > > > >> >> can introduce it to other window-related
> operators
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> in the future.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> 3. You need to modify the
> > > > > > FlinkChangelogModeInferenceProgram
> > > > > > > to
> > > > > > > > > > > ensure
> > > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> early firing enabled are aware of retract or
> upsert
> > > > > > messages.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> --
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> Best！
> > > > > > > > > > > >> >> Xuyang
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > > xingcanc@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire
> > for
> > > > time
> > > > > > > > > interval
> > > > > > > > > > > outer
> > > > > > > > > > > >> >> >joins is feasible overall. I have a few
> questions.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >1. The current output stream of a time interval
> > > outer
> > > > > join
> > > > > > > is
> > > > > > > > an
> > > > > > > > > > > >> >> >append-only stream. This change will make it a
> > > > potential
> > > > > > > > > > retractable
> > > > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> > > dynamic
> > > > > > > output
> > > > > > > > > type
> > > > > > > > > > > like
> > > > > > > > > > > >> >> >that. Could you add this part to your design
> doc?
> > > > > > > > > > > >> >> >2. What's the use case when the downstream
> > > components
> > > > > need
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > the
> > > > > > > > > > > >> >> early
> > > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > > >> >> >3. The time interval join operator itself is not
> > > quite
> > > > > > > > efficient
> > > > > > > > > > > when
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > > overhead
> > > > > > after
> > > > > > > > > > > introducing
> > > > > > > > > > > >> >> >this feature?
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > > >> >> >wrote:
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Just a gentle reminder regarding the proposal
> I
> > > > shared
> > > > > > on
> > > > > > > > > early
> > > > > > > > > > > fire
> > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> > greatly
> > > > > > > appreciate
> > > > > > > > > > your
> > > > > > > > > > > >> >> feedback
> > > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > > > > > >> >> >> <
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Best,
> > > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang
> <
> > > > > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > > introducing
> > > > > > > early
> > > > > > > > > fire
> > > > > > > > > > > >> support
> > > > > > > > > > > >> >> >> for
> > > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > > >> >> >> > In many streaming applications, particularly
> > > > > real-time
> > > > > > > > > > analytics
> > > > > > > > > > > >> and
> > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to obtain
> > > > partial
> > > > > > > > results
> > > > > > > > > > > >> earlier
> > > > > > > > > > > >> >> >> rather
> > > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > > finalize.
> > > > > For
> > > > > > > > Flink
> > > > > > > > > > SQL
> > > > > > > > > > > >> >> interval
> > > > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > > > watermarks
> > > > > > > > ensure
> > > > > > > > > > no
> > > > > > > > > > > >> more
> > > > > > > > > > > >> >> >> > matches can occur. This delay can be
> > challenging
> > > > for
> > > > > > > > > scenarios
> > > > > > > > > > > that
> > > > > > > > > > > >> >> >> require
> > > > > > > > > > > >> >> >> > fast feedback. Early fire support addresses
> > this
> > > > by
> > > > > > > > emitting
> > > > > > > > > > > >> >> intermediate
> > > > > > > > > > > >> >> >> > results speculatively and using retractions
> or
> > > > > updates
> > > > > > > to
> > > > > > > > > > > maintain
> > > > > > > > > > > >> >> >> eventual
> > > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > > >> >> >> > <
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
> refine
> > > this
> > > > > > > > feature.
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > > >> >> >> >
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Becket Qin <be...@gmail.com>.
Sorry for the late reply. I have a question regarding the configuration of
interval. I am wondering in which scenario the interval config will
actually be used. Is the interval configuration only useful for windowed
aggregation?
For the join operations, early fire seems only applicable to the outer join
before there is a match. Once there is a match, the output of the join
operation should be just event driven. For example, consider the following
scenario:
1. left outer join, and there is no match from the right side before the
initial delay
2. a record [left, null] was emitted due to early fire
3. one of the following two cases must happen
a. there is still no match event from the right side before the
specific interval passes. In this case, we are not supposed to emit another
[left, null].
b. if there is a match, a record of [left, right] should be emitted
immediately regardless of the interval. If this happens, the interval will
be ignored from this point on.
That said, if the configuration proposed in this FLIP is intended not only
for join, but for general purpose early fire, then the interval config
makes sense.
Thanks,
Jiangjie (Becket) Qin
On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Thank you all for reviewing. As there are no objections, I’ll move forward
> with a vote.
>
> Best regards,
> Weiqing
>
> On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> vsowrira@asu.edu>
> wrote:
>
> > Same here, I don't have any more questions. Thanks!
> >
> > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com> wrote:
> >
> > > Hi Weiqing,
> > >
> > > I don't have any more questions. The doc looks good to me.
> > >
> > > Thanks,
> > > Xingcan
> > >
> > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > vsowrira@asu.edu>
> > > wrote:
> > >
> > > > Hi Weiqing,
> > > >
> > > > Thanks, that makes sense! Looks like I missed it.
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > yangweiqing001@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Venkata,
> > > > >
> > > > > * > where only one earlyFire is fired The DELAY *
> > > > >
> > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > >
> > > > > of the proposal can produce a single early fire per interval,
> > aligning
> > > > with
> > > > > your suggestion. How it integrates with existing early-fire
> > > > configurations
> > > > > are mentioned here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > >.
> > > > > Let me know if you have any further questions or feedback!
> > > > >
> > > > > Thanks,
> > > > > Weiqing
> > > > >
> > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > vsowrira@asu.edu> wrote:
> > > > >
> > > > > > Thanks for the response!
> > > > > >
> > > > > > > If the optional `interval` in the proposal is enabled,
> early-fire
> > > > > outputs
> > > > > > will occur repeatedly at every earlyFireInterval, not just once.
> > > After
> > > > > that
> > > > > > interval concludes, there will also be a final emission.
> > > > > >
> > > > > > One more question, would it make sense to support a mechanism
> > through
> > > > > > configuration where only one earlyFire is fired for an interval
> or
> > > > > window?
> > > > > > This would also simplify the state overhead for use cases where
> > only
> > > > one
> > > > > > early fire is required within a window or interval. Thoughts?
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > yangweiqing001@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Venkata,
> > > > > > >
> > > > > > > Thanks for your feedback!
> > > > > > >
> > > > > > > If the optional `interval` in the proposal is enabled,
> early-fire
> > > > > outputs
> > > > > > > will occur repeatedly at every earlyFireInterval, not just
> once.
> > > > After
> > > > > > that
> > > > > > > interval concludes, there will also be a final emission.
> > > > > > >
> > > > > > > We haven’t included a late-fire mechanism for interval joins in
> > > this
> > > > > > FLIP,
> > > > > > > but it’s certainly something we can consider for future
> efforts!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Weiqing
> > > > > > >
> > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > > > > vsowrira@asu.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Weiqing,
> > > > > > > >
> > > > > > > > Thanks, this is a great addition to Flink SQL. Also instead
> of
> > > > > > > controlling
> > > > > > > > and configuring through Flink configs unlike the older window
> > > > > > > aggregation,
> > > > > > > > hints seems to be a much better approach. This enables a
> > > > customizable
> > > > > > > early
> > > > > > > > fire behavior for individual interval joins.
> > > > > > > >
> > > > > > > > Couple of questions:
> > > > > > > >
> > > > > > > > 1. Does the *early fire* emit an output every
> earlyFireInterval
> > > > time
> > > > > or
> > > > > > > > will it be a one time output emission and another output
> > emitted
> > > at
> > > > > the
> > > > > > > end
> > > > > > > > of the interval?
> > > > > > > > 2. Are there plans to support *late fire *similar to the
> window
> > > > > > > > aggregations in later FLIPs?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Venkata krishnan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > yangweiqing001@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > >
> > > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > > > > > > >
> > > > > > > > > If no further objections arise from anyone, I’ll proceed to
> > > mark
> > > > > FLIP
> > > > > > > as
> > > > > > > > > ready for voting.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Weiqing
> > > > > > > > >
> > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xyzhong131@163.com
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > LGTM overall. Thanks for updating. I have no problem and
> +1
> > > for
> > > > > > this
> > > > > > > > > > feature.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Best！
> > > > > > > > > > Xuyang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > yangweiqing001@gmail.com
> > > > >
> > > > > > 写道：
> > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > >
> > > > > > > > > > >Thank you for your detailed feedback! I’ve updated the
> > > > proposal
> > > > > > doc
> > > > > > > > > > >accordingly. Please feel free to take another look and
> let
> > > me
> > > > > know
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > >
> > > > > > > > > > >Best regards,
> > > > > > > > > > >Weiqing
> > > > > > > > > > >
> > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > xyzhong131@163.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > >>
> > > > > > > > > > >> After reading the new FLIP, I have no issues with the
> > part
> > > > > > `public
> > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > >>
> > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > IntervalJoin
> > > > > > node,
> > > > > > > > > IIUC:
> > > > > > > > > > >>
> > > > > > > > > > >> - When early firing is enabled, the UpdateKind of the
> > > > > > IntervalJoin
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > >>
> > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > > > > > requirements
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > >>
> > > > > > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > > > > > IntervalJoin
> > > > > > > > is
> > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > >>
> > > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > > disabled,
> > > > > the
> > > > > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > > > > >>
> > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >>
> > > > > > > > > > >> Best！
> > > > > > > > > > >> Xuyang
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > >> >
> > > > > > > > > > >> >Thanks so much for the feedback - it was very
> helpful!
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 1. The current output stream of a time interval
> > outer
> > > > join
> > > > > > is
> > > > > > > an
> > > > > > > > > > >> >append-only stream. This change will make it a
> > potential
> > > > > > > > retractable
> > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> dynamic
> > > > output
> > > > > > > type
> > > > > > > > > like
> > > > > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > - Yes, enabling early firing on time interval
> outer
> > > > joins
> > > > > > can
> > > > > > > > emit
> > > > > > > > > > >> > retractions when previously emitted rows are
> updated
> > > or
> > > > > > > > > invalidated
> > > > > > > > > > by
> > > > > > > > > > >> > later matches. I’ve updated the proposal (Planner
> > > > > Awareness
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > >> >
> > > > > > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > >> >)
> > > > > > > > > > >> > to clarify that the stream might switch from
> > > append-only
> > > > > to
> > > > > > a
> > > > > > > > > > >> > retract/upsert stream. Let me know if anything is
> > > > missing.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > components
> > > > need
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > - The new INTERVAL option (in addition to DELAY)
> > > allows
> > > > > > > periodic
> > > > > > > > > > >> updates
> > > > > > > > > > >> > (e.g., every 10 minutes) after the initial delay.
> > This
> > > > > > > captures
> > > > > > > > > how
> > > > > > > > > > >> results
> > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > “Repeatedly”
> > > > > > > option.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 3. The time interval join operator itself is not
> > quite
> > > > > > > efficient
> > > > > > > > > > when
> > > > > > > > > > >> >the state becomes large. Will there be any extra
> > overhead
> > > > > after
> > > > > > > > > > >> introducing
> > > > > > > > > > >> >this feature?*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > potentially
> > > > > > > > emitting
> > > > > > > > > > >> > partial matches multiple times with retraction
> > > (avoiding
> > > > > > > > duplicate
> > > > > > > > > > >> outputs
> > > > > > > > > > >> > though). However, if it’s disabled, there is no
> > > > additional
> > > > > > > cost.
> > > > > > > > > > Most
> > > > > > > > > > >> users
> > > > > > > > > > >> > find the performance trade-off acceptable for the
> > > > > real-time
> > > > > > > > > > insights it
> > > > > > > > > > >> > provides.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 1. Currently, there are some configs related to
> > early
> > > > > firing
> > > > > > > > > > available
> > > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > >> >**lay`.
> > > > > > > > > > >> >Although their documentation states that they are
> only
> > > > > > applicable
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > >> >Window operator, it seems possible that they may also
> > be
> > > > > > relevant
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >> >context of this FLIP. Otherwise, having different
> early
> > > > > firing
> > > > > > > > > > behaviors
> > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > > confusion.
> > > > > > I’ve
> > > > > > > > > > added a
> > > > > > > > > > >> > section
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > >> >
> > > > > > > > > > >> >in
> > > > > > > > > > >> > the proposal highlighting that we should align
> > > > hint-based
> > > > > > > > interval
> > > > > > > > > > join
> > > > > > > > > > >> > configurations with the existing table.exec.emit.*
> > > > > settings.
> > > > > > > > > > >> Suggestions
> > > > > > > > > > >> > on how to make the unification are welcome! We
> plan
> > to
> > > > > > extend
> > > > > > > > > early
> > > > > > > > > > >> firing
> > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar
> > to
> > > > > point
> > > > > > 1,
> > > > > > > > > > perhaps
> > > > > > > > > > >> >we can introduce it to other window-related operators
> > in
> > > > the
> > > > > > > > future.>
> > > > > > > > > > 3.
> > > > > > > > > > >> >You need to modify the
> > FlinkChangelogModeInferenceProgram
> > > > to
> > > > > > > ensure
> > > > > > > > > > that
> > > > > > > > > > >> >downstream nodes of interval joins with early firing
> > > > enabled
> > > > > > are
> > > > > > > > > aware
> > > > > > > > > > of
> > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - We agree that time_mode could be introduced to
> > other
> > > > > > > > > window-based
> > > > > > > > > > >> > operators down the road. We also want to support
> > early
> > > > > fire
> > > > > > > for
> > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the
> code
> > > > > change
> > > > > > on
> > > > > > > > it
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > >> >
> > > > > > > > > > >> > in the proposal.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >Thanks again for your time and feedback! I’ve updated
> > the
> > > > > > > proposal
> > > > > > > > > with
> > > > > > > > > > >> >these points. Please let me know if there’s anything
> > > else I
> > > > > > > should
> > > > > > > > > > >> address.
> > > > > > > > > > >> >
> > > > > > > > > > >> >Best,
> > > > > > > > > > >> >Weiqing
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > xyzhong131@163.com>
> > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I
> > have a
> > > > few
> > > > > > > > > > questions:
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 1. Currently, there are some configs related to
> early
> > > > > firing
> > > > > > > > > > available
> > > > > > > > > > >> to
> > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > > > > > documentation
> > > > > > > > > > states
> > > > > > > > > > >> >> that they are only applicable to the Window
> operator,
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> it seems possible that they may also be relevant in
> > the
> > > > > > context
> > > > > > > > of
> > > > > > > > > > this
> > > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > > behaviors
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 2. The design of `time_mode` is excellent. Similar
> to
> > > > point
> > > > > > 1,
> > > > > > > > > > perhaps
> > > > > > > > > > >> we
> > > > > > > > > > >> >> can introduce it to other window-related operators
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> in the future.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 3. You need to modify the
> > > > > FlinkChangelogModeInferenceProgram
> > > > > > to
> > > > > > > > > > ensure
> > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> early firing enabled are aware of retract or upsert
> > > > > messages.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> --
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Best！
> > > > > > > > > > >> >> Xuyang
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > xingcanc@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire
> for
> > > time
> > > > > > > > interval
> > > > > > > > > > outer
> > > > > > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >1. The current output stream of a time interval
> > outer
> > > > join
> > > > > > is
> > > > > > > an
> > > > > > > > > > >> >> >append-only stream. This change will make it a
> > > potential
> > > > > > > > > retractable
> > > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> > dynamic
> > > > > > output
> > > > > > > > type
> > > > > > > > > > like
> > > > > > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > > > > > >> >> >2. What's the use case when the downstream
> > components
> > > > need
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > >> >> early
> > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > >> >> >3. The time interval join operator itself is not
> > quite
> > > > > > > efficient
> > > > > > > > > > when
> > > > > > > > > > >> the
> > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > overhead
> > > > > after
> > > > > > > > > > introducing
> > > > > > > > > > >> >> >this feature?
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > > >> >> >wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Just a gentle reminder regarding the proposal I
> > > shared
> > > > > on
> > > > > > > > early
> > > > > > > > > > fire
> > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> greatly
> > > > > > appreciate
> > > > > > > > > your
> > > > > > > > > > >> >> feedback
> > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > > > > >> >> >> <
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Best,
> > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > introducing
> > > > > > early
> > > > > > > > fire
> > > > > > > > > > >> support
> > > > > > > > > > >> >> >> for
> > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > >> >> >> > In many streaming applications, particularly
> > > > real-time
> > > > > > > > > analytics
> > > > > > > > > > >> and
> > > > > > > > > > >> >> >> > monitoring systems, it is valuable to obtain
> > > partial
> > > > > > > results
> > > > > > > > > > >> earlier
> > > > > > > > > > >> >> >> rather
> > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > finalize.
> > > > For
> > > > > > > Flink
> > > > > > > > > SQL
> > > > > > > > > > >> >> interval
> > > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > > watermarks
> > > > > > > ensure
> > > > > > > > > no
> > > > > > > > > > >> more
> > > > > > > > > > >> >> >> > matches can occur. This delay can be
> challenging
> > > for
> > > > > > > > scenarios
> > > > > > > > > > that
> > > > > > > > > > >> >> >> require
> > > > > > > > > > >> >> >> > fast feedback. Early fire support addresses
> this
> > > by
> > > > > > > emitting
> > > > > > > > > > >> >> intermediate
> > > > > > > > > > >> >> >> > results speculatively and using retractions or
> > > > updates
> > > > > > to
> > > > > > > > > > maintain
> > > > > > > > > > >> >> >> eventual
> > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > >> >> >> > <
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to refine
> > this
> > > > > > > feature.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Thank you all for reviewing. As there are no objections, I’ll move forward
with a vote.
Best regards,
Weiqing
On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <vs...@asu.edu>
wrote:
> Same here, I don't have any more questions. Thanks!
>
> On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com> wrote:
>
> > Hi Weiqing,
> >
> > I don't have any more questions. The doc looks good to me.
> >
> > Thanks,
> > Xingcan
> >
> > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > vsowrira@asu.edu>
> > wrote:
> >
> > > Hi Weiqing,
> > >
> > > Thanks, that makes sense! Looks like I missed it.
> > >
> > > Regards
> > > Venkata krishnan
> > >
> > >
> > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> yangweiqing001@gmail.com>
> > > wrote:
> > >
> > > > Hi Venkata,
> > > >
> > > > * > where only one earlyFire is fired The DELAY *
> > > >
> > > > The DELAY option mentioned in the Public Interfaces section
> > > > <
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > >
> > > > of the proposal can produce a single early fire per interval,
> aligning
> > > with
> > > > your suggestion. How it integrates with existing early-fire
> > > configurations
> > > > are mentioned here
> > > > <
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > >.
> > > > Let me know if you have any further questions or feedback!
> > > >
> > > > Thanks,
> > > > Weiqing
> > > >
> > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > vsowrira@asu.edu> wrote:
> > > >
> > > > > Thanks for the response!
> > > > >
> > > > > > If the optional `interval` in the proposal is enabled, early-fire
> > > > outputs
> > > > > will occur repeatedly at every earlyFireInterval, not just once.
> > After
> > > > that
> > > > > interval concludes, there will also be a final emission.
> > > > >
> > > > > One more question, would it make sense to support a mechanism
> through
> > > > > configuration where only one earlyFire is fired for an interval or
> > > > window?
> > > > > This would also simplify the state overhead for use cases where
> only
> > > one
> > > > > early fire is required within a window or interval. Thoughts?
> > > > >
> > > > > Regards
> > > > > Venkata krishnan
> > > > >
> > > > >
> > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > yangweiqing001@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Venkata,
> > > > > >
> > > > > > Thanks for your feedback!
> > > > > >
> > > > > > If the optional `interval` in the proposal is enabled, early-fire
> > > > outputs
> > > > > > will occur repeatedly at every earlyFireInterval, not just once.
> > > After
> > > > > that
> > > > > > interval concludes, there will also be a final emission.
> > > > > >
> > > > > > We haven’t included a late-fire mechanism for interval joins in
> > this
> > > > > FLIP,
> > > > > > but it’s certainly something we can consider for future efforts!
> > > > > >
> > > > > > Best regards,
> > > > > > Weiqing
> > > > > >
> > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > > > vsowrira@asu.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Weiqing,
> > > > > > >
> > > > > > > Thanks, this is a great addition to Flink SQL. Also instead of
> > > > > > controlling
> > > > > > > and configuring through Flink configs unlike the older window
> > > > > > aggregation,
> > > > > > > hints seems to be a much better approach. This enables a
> > > customizable
> > > > > > early
> > > > > > > fire behavior for individual interval joins.
> > > > > > >
> > > > > > > Couple of questions:
> > > > > > >
> > > > > > > 1. Does the *early fire* emit an output every earlyFireInterval
> > > time
> > > > or
> > > > > > > will it be a one time output emission and another output
> emitted
> > at
> > > > the
> > > > > > end
> > > > > > > of the interval?
> > > > > > > 2. Are there plans to support *late fire *similar to the window
> > > > > > > aggregations in later FLIPs?
> > > > > > >
> > > > > > > Regards
> > > > > > > Venkata krishnan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > >
> > > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > > > > > >
> > > > > > > > If no further objections arise from anyone, I’ll proceed to
> > mark
> > > > FLIP
> > > > > > as
> > > > > > > > ready for voting.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Weiqing
> > > > > > > >
> > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > LGTM overall. Thanks for updating. I have no problem and +1
> > for
> > > > > this
> > > > > > > > > feature.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best！
> > > > > > > > > Xuyang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > yangweiqing001@gmail.com
> > > >
> > > > > 写道：
> > > > > > > > > >Hi Xuyang,
> > > > > > > > > >
> > > > > > > > > >Thank you for your detailed feedback! I’ve updated the
> > > proposal
> > > > > doc
> > > > > > > > > >accordingly. Please feel free to take another look and let
> > me
> > > > know
> > > > > > if
> > > > > > > > you
> > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > >
> > > > > > > > > >Best regards,
> > > > > > > > > >Weiqing
> > > > > > > > > >
> > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> xyzhong131@163.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > >>
> > > > > > > > > >> After reading the new FLIP, I have no issues with the
> part
> > > > > `public
> > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > >>
> > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > >>
> > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > IntervalJoin
> > > > > node,
> > > > > > > > IIUC:
> > > > > > > > > >>
> > > > > > > > > >> - When early firing is enabled, the UpdateKind of the
> > > > > IntervalJoin
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > >>
> > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > > > > requirements
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > >>
> > > > > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > > > > IntervalJoin
> > > > > > > is
> > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > >>
> > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > disabled,
> > > > the
> > > > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > > > >>
> > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >>
> > > > > > > > > >> Best！
> > > > > > > > > >> Xuyang
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > yangweiqing001@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > >> >
> > > > > > > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > > > > > > >> >
> > > > > > > > > >> >*> 1. The current output stream of a time interval
> outer
> > > join
> > > > > is
> > > > > > an
> > > > > > > > > >> >append-only stream. This change will make it a
> potential
> > > > > > > retractable
> > > > > > > > > >> >stream. I'm not sure if the planner supports a dynamic
> > > output
> > > > > > type
> > > > > > > > like
> > > > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > - Yes, enabling early firing on time interval outer
> > > joins
> > > > > can
> > > > > > > emit
> > > > > > > > > >> > retractions when previously emitted rows are updated
> > or
> > > > > > > > invalidated
> > > > > > > > > by
> > > > > > > > > >> > later matches. I’ve updated the proposal (Planner
> > > > Awareness
> > > > > > > > > >> > <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > >> >
> > > > > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > > > > >> > <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > >> >)
> > > > > > > > > >> > to clarify that the stream might switch from
> > append-only
> > > > to
> > > > > a
> > > > > > > > > >> > retract/upsert stream. Let me know if anything is
> > > missing.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >*> 2. What's the use case when the downstream
> components
> > > need
> > > > > to
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > - The new INTERVAL option (in addition to DELAY)
> > allows
> > > > > > periodic
> > > > > > > > > >> updates
> > > > > > > > > >> > (e.g., every 10 minutes) after the initial delay.
> This
> > > > > > captures
> > > > > > > > how
> > > > > > > > > >> results
> > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > “Repeatedly”
> > > > > > option.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >*> 3. The time interval join operator itself is not
> quite
> > > > > > efficient
> > > > > > > > > when
> > > > > > > > > >> >the state becomes large. Will there be any extra
> overhead
> > > > after
> > > > > > > > > >> introducing
> > > > > > > > > >> >this feature?*
> > > > > > > > > >> >
> > > > > > > > > >> > - Early fire does introduce some overhead by
> > potentially
> > > > > > > emitting
> > > > > > > > > >> > partial matches multiple times with retraction
> > (avoiding
> > > > > > > duplicate
> > > > > > > > > >> outputs
> > > > > > > > > >> > though). However, if it’s disabled, there is no
> > > additional
> > > > > > cost.
> > > > > > > > > Most
> > > > > > > > > >> users
> > > > > > > > > >> > find the performance trade-off acceptable for the
> > > > real-time
> > > > > > > > > insights it
> > > > > > > > > >> > provides.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >*> 1. Currently, there are some configs related to
> early
> > > > firing
> > > > > > > > > available
> > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > >> >**lay`.
> > > > > > > > > >> >Although their documentation states that they are only
> > > > > applicable
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > >> >Window operator, it seems possible that they may also
> be
> > > > > relevant
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >> >context of this FLIP. Otherwise, having different early
> > > > firing
> > > > > > > > > behaviors
> > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > >> >
> > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > confusion.
> > > > > I’ve
> > > > > > > > > added a
> > > > > > > > > >> > section
> > > > > > > > > >> > <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > >> >
> > > > > > > > > >> >in
> > > > > > > > > >> > the proposal highlighting that we should align
> > > hint-based
> > > > > > > interval
> > > > > > > > > join
> > > > > > > > > >> > configurations with the existing table.exec.emit.*
> > > > settings.
> > > > > > > > > >> Suggestions
> > > > > > > > > >> > on how to make the unification are welcome! We plan
> to
> > > > > extend
> > > > > > > > early
> > > > > > > > > >> firing
> > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar
> to
> > > > point
> > > > > 1,
> > > > > > > > > perhaps
> > > > > > > > > >> >we can introduce it to other window-related operators
> in
> > > the
> > > > > > > future.>
> > > > > > > > > 3.
> > > > > > > > > >> >You need to modify the
> FlinkChangelogModeInferenceProgram
> > > to
> > > > > > ensure
> > > > > > > > > that
> > > > > > > > > >> >downstream nodes of interval joins with early firing
> > > enabled
> > > > > are
> > > > > > > > aware
> > > > > > > > > of
> > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > >> >
> > > > > > > > > >> > - We agree that time_mode could be introduced to
> other
> > > > > > > > window-based
> > > > > > > > > >> > operators down the road. We also want to support
> early
> > > > fire
> > > > > > for
> > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the code
> > > > change
> > > > > on
> > > > > > > it
> > > > > > > > > >> > <
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > >> >
> > > > > > > > > >> > in the proposal.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >Thanks again for your time and feedback! I’ve updated
> the
> > > > > > proposal
> > > > > > > > with
> > > > > > > > > >> >these points. Please let me know if there’s anything
> > else I
> > > > > > should
> > > > > > > > > >> address.
> > > > > > > > > >> >
> > > > > > > > > >> >Best,
> > > > > > > > > >> >Weiqing
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > xyzhong131@163.com>
> > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I
> have a
> > > few
> > > > > > > > > questions:
> > > > > > > > > >> >>
> > > > > > > > > >> >> 1. Currently, there are some configs related to early
> > > > firing
> > > > > > > > > available
> > > > > > > > > >> to
> > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > >> >>
> > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > > > > documentation
> > > > > > > > > states
> > > > > > > > > >> >> that they are only applicable to the Window operator,
> > > > > > > > > >> >>
> > > > > > > > > >> >> it seems possible that they may also be relevant in
> the
> > > > > context
> > > > > > > of
> > > > > > > > > this
> > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > behaviors
> > > > > > > > > >> >>
> > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > >> >>
> > > > > > > > > >> >> 2. The design of `time_mode` is excellent. Similar to
> > > point
> > > > > 1,
> > > > > > > > > perhaps
> > > > > > > > > >> we
> > > > > > > > > >> >> can introduce it to other window-related operators
> > > > > > > > > >> >>
> > > > > > > > > >> >> in the future.
> > > > > > > > > >> >>
> > > > > > > > > >> >> 3. You need to modify the
> > > > FlinkChangelogModeInferenceProgram
> > > > > to
> > > > > > > > > ensure
> > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > >> >>
> > > > > > > > > >> >> early firing enabled are aware of retract or upsert
> > > > messages.
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >> --
> > > > > > > > > >> >>
> > > > > > > > > >> >> Best！
> > > > > > > > > >> >> Xuyang
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > xingcanc@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire for
> > time
> > > > > > > interval
> > > > > > > > > outer
> > > > > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >1. The current output stream of a time interval
> outer
> > > join
> > > > > is
> > > > > > an
> > > > > > > > > >> >> >append-only stream. This change will make it a
> > potential
> > > > > > > > retractable
> > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> dynamic
> > > > > output
> > > > > > > type
> > > > > > > > > like
> > > > > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > > > > >> >> >2. What's the use case when the downstream
> components
> > > need
> > > > > to
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > >> >> early
> > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > >> >> >3. The time interval join operator itself is not
> quite
> > > > > > efficient
> > > > > > > > > when
> > > > > > > > > >> the
> > > > > > > > > >> >> >state becomes large. Will there be any extra
> overhead
> > > > after
> > > > > > > > > introducing
> > > > > > > > > >> >> >this feature?
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >Thanks,
> > > > > > > > > >> >> >Xingcan
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > > >> >> >wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> Hi all,
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Just a gentle reminder regarding the proposal I
> > shared
> > > > on
> > > > > > > early
> > > > > > > > > fire
> > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d greatly
> > > > > appreciate
> > > > > > > > your
> > > > > > > > > >> >> feedback
> > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > > > >> >> >> <
> > > > > > > > > >> >> >>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Thank you!
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Best,
> > > > > > > > > >> >> >> Weiqing
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> wrote:
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > introducing
> > > > > early
> > > > > > > fire
> > > > > > > > > >> support
> > > > > > > > > >> >> >> for
> > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > >> >> >> > In many streaming applications, particularly
> > > real-time
> > > > > > > > analytics
> > > > > > > > > >> and
> > > > > > > > > >> >> >> > monitoring systems, it is valuable to obtain
> > partial
> > > > > > results
> > > > > > > > > >> earlier
> > > > > > > > > >> >> >> rather
> > > > > > > > > >> >> >> > than waiting for full join conditions to
> finalize.
> > > For
> > > > > > Flink
> > > > > > > > SQL
> > > > > > > > > >> >> interval
> > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > watermarks
> > > > > > ensure
> > > > > > > > no
> > > > > > > > > >> more
> > > > > > > > > >> >> >> > matches can occur. This delay can be challenging
> > for
> > > > > > > scenarios
> > > > > > > > > that
> > > > > > > > > >> >> >> require
> > > > > > > > > >> >> >> > fast feedback. Early fire support addresses this
> > by
> > > > > > emitting
> > > > > > > > > >> >> intermediate
> > > > > > > > > >> >> >> > results speculatively and using retractions or
> > > updates
> > > > > to
> > > > > > > > > maintain
> > > > > > > > > >> >> >> eventual
> > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > >> >> >> > <
> > > > > > > > > >> >> >>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> > Your feedback and ideas are welcome to refine
> this
> > > > > > feature.
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > >> >> >> >
> > > > > > > > > >> >> >>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Same here, I don't have any more questions. Thanks!
On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xi...@gmail.com> wrote:
> Hi Weiqing,
>
> I don't have any more questions. The doc looks good to me.
>
> Thanks,
> Xingcan
>
> On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> vsowrira@asu.edu>
> wrote:
>
> > Hi Weiqing,
> >
> > Thanks, that makes sense! Looks like I missed it.
> >
> > Regards
> > Venkata krishnan
> >
> >
> > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <ya...@gmail.com>
> > wrote:
> >
> > > Hi Venkata,
> > >
> > > * > where only one earlyFire is fired The DELAY *
> > >
> > > The DELAY option mentioned in the Public Interfaces section
> > > <
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > >
> > > of the proposal can produce a single early fire per interval, aligning
> > with
> > > your suggestion. How it integrates with existing early-fire
> > configurations
> > > are mentioned here
> > > <
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > >.
> > > Let me know if you have any further questions or feedback!
> > >
> > > Thanks,
> > > Weiqing
> > >
> > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > vsowrira@asu.edu> wrote:
> > >
> > > > Thanks for the response!
> > > >
> > > > > If the optional `interval` in the proposal is enabled, early-fire
> > > outputs
> > > > will occur repeatedly at every earlyFireInterval, not just once.
> After
> > > that
> > > > interval concludes, there will also be a final emission.
> > > >
> > > > One more question, would it make sense to support a mechanism through
> > > > configuration where only one earlyFire is fired for an interval or
> > > window?
> > > > This would also simplify the state overhead for use cases where only
> > one
> > > > early fire is required within a window or interval. Thoughts?
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> yangweiqing001@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Venkata,
> > > > >
> > > > > Thanks for your feedback!
> > > > >
> > > > > If the optional `interval` in the proposal is enabled, early-fire
> > > outputs
> > > > > will occur repeatedly at every earlyFireInterval, not just once.
> > After
> > > > that
> > > > > interval concludes, there will also be a final emission.
> > > > >
> > > > > We haven’t included a late-fire mechanism for interval joins in
> this
> > > > FLIP,
> > > > > but it’s certainly something we can consider for future efforts!
> > > > >
> > > > > Best regards,
> > > > > Weiqing
> > > > >
> > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > > vsowrira@asu.edu>
> > > > > wrote:
> > > > >
> > > > > > Weiqing,
> > > > > >
> > > > > > Thanks, this is a great addition to Flink SQL. Also instead of
> > > > > controlling
> > > > > > and configuring through Flink configs unlike the older window
> > > > > aggregation,
> > > > > > hints seems to be a much better approach. This enables a
> > customizable
> > > > > early
> > > > > > fire behavior for individual interval joins.
> > > > > >
> > > > > > Couple of questions:
> > > > > >
> > > > > > 1. Does the *early fire* emit an output every earlyFireInterval
> > time
> > > or
> > > > > > will it be a one time output emission and another output emitted
> at
> > > the
> > > > > end
> > > > > > of the interval?
> > > > > > 2. Are there plans to support *late fire *similar to the window
> > > > > > aggregations in later FLIPs?
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > yangweiqing001@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for reviewing, Xuyang!
> > > > > > >
> > > > > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > > > > >
> > > > > > > If no further objections arise from anyone, I’ll proceed to
> mark
> > > FLIP
> > > > > as
> > > > > > > ready for voting.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Weiqing
> > > > > > >
> > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com>
> > wrote:
> > > > > > >
> > > > > > > > LGTM overall. Thanks for updating. I have no problem and +1
> for
> > > > this
> > > > > > > > feature.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best！
> > > > > > > > Xuyang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> yangweiqing001@gmail.com
> > >
> > > > 写道：
> > > > > > > > >Hi Xuyang,
> > > > > > > > >
> > > > > > > > >Thank you for your detailed feedback! I’ve updated the
> > proposal
> > > > doc
> > > > > > > > >accordingly. Please feel free to take another look and let
> me
> > > know
> > > > > if
> > > > > > > you
> > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > >
> > > > > > > > >Best regards,
> > > > > > > > >Weiqing
> > > > > > > > >
> > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi, Weiqing.
> > > > > > > > >>
> > > > > > > > >> After reading the new FLIP, I have no issues with the part
> > > > `public
> > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > >>
> > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > >>
> > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> IntervalJoin
> > > > node,
> > > > > > > IIUC:
> > > > > > > > >>
> > > > > > > > >> - When early firing is enabled, the UpdateKind of the
> > > > IntervalJoin
> > > > > > can
> > > > > > > > be
> > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > >>
> > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > > > requirements
> > > > > > of
> > > > > > > > the
> > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > >>
> > > > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > > > IntervalJoin
> > > > > > is
> > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > >>
> > > > > > > > >> - Nevertheless, whether early firing is enabled or
> disabled,
> > > the
> > > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > > >>
> > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >>
> > > > > > > > >> Best！
> > > > > > > > >> Xuyang
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > yangweiqing001@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > >> >
> > > > > > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > > > > > >> >
> > > > > > > > >> >*> 1. The current output stream of a time interval outer
> > join
> > > > is
> > > > > an
> > > > > > > > >> >append-only stream. This change will make it a potential
> > > > > > retractable
> > > > > > > > >> >stream. I'm not sure if the planner supports a dynamic
> > output
> > > > > type
> > > > > > > like
> > > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > - Yes, enabling early firing on time interval outer
> > joins
> > > > can
> > > > > > emit
> > > > > > > > >> > retractions when previously emitted rows are updated
> or
> > > > > > > invalidated
> > > > > > > > by
> > > > > > > > >> > later matches. I’ve updated the proposal (Planner
> > > Awareness
> > > > > > > > >> > <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > >> >
> > > > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > > > >> > <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > >> >)
> > > > > > > > >> > to clarify that the stream might switch from
> append-only
> > > to
> > > > a
> > > > > > > > >> > retract/upsert stream. Let me know if anything is
> > missing.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >*> 2. What's the use case when the downstream components
> > need
> > > > to
> > > > > > get
> > > > > > > > the
> > > > > > > > >> >early fired results regularly?*
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > - The new INTERVAL option (in addition to DELAY)
> allows
> > > > > periodic
> > > > > > > > >> updates
> > > > > > > > >> > (e.g., every 10 minutes) after the initial delay. This
> > > > > captures
> > > > > > > how
> > > > > > > > >> results
> > > > > > > > >> > evolve over time, similar to Apache Beam’s
> “Repeatedly”
> > > > > option.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >*> 3. The time interval join operator itself is not quite
> > > > > efficient
> > > > > > > > when
> > > > > > > > >> >the state becomes large. Will there be any extra overhead
> > > after
> > > > > > > > >> introducing
> > > > > > > > >> >this feature?*
> > > > > > > > >> >
> > > > > > > > >> > - Early fire does introduce some overhead by
> potentially
> > > > > > emitting
> > > > > > > > >> > partial matches multiple times with retraction
> (avoiding
> > > > > > duplicate
> > > > > > > > >> outputs
> > > > > > > > >> > though). However, if it’s disabled, there is no
> > additional
> > > > > cost.
> > > > > > > > Most
> > > > > > > > >> users
> > > > > > > > >> > find the performance trade-off acceptable for the
> > > real-time
> > > > > > > > insights it
> > > > > > > > >> > provides.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >*> 1. Currently, there are some configs related to early
> > > firing
> > > > > > > > available
> > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > >> >**lay`.
> > > > > > > > >> >Although their documentation states that they are only
> > > > applicable
> > > > > > to
> > > > > > > > the
> > > > > > > > >> >Window operator, it seems possible that they may also be
> > > > relevant
> > > > > > in
> > > > > > > > the
> > > > > > > > >> >context of this FLIP. Otherwise, having different early
> > > firing
> > > > > > > > behaviors
> > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > >> >
> > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > confusion.
> > > > I’ve
> > > > > > > > added a
> > > > > > > > >> > section
> > > > > > > > >> > <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > >> >
> > > > > > > > >> >in
> > > > > > > > >> > the proposal highlighting that we should align
> > hint-based
> > > > > > interval
> > > > > > > > join
> > > > > > > > >> > configurations with the existing table.exec.emit.*
> > > settings.
> > > > > > > > >> Suggestions
> > > > > > > > >> > on how to make the unification are welcome! We plan to
> > > > extend
> > > > > > > early
> > > > > > > > >> firing
> > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar to
> > > point
> > > > 1,
> > > > > > > > perhaps
> > > > > > > > >> >we can introduce it to other window-related operators in
> > the
> > > > > > future.>
> > > > > > > > 3.
> > > > > > > > >> >You need to modify the FlinkChangelogModeInferenceProgram
> > to
> > > > > ensure
> > > > > > > > that
> > > > > > > > >> >downstream nodes of interval joins with early firing
> > enabled
> > > > are
> > > > > > > aware
> > > > > > > > of
> > > > > > > > >> >retract or upsert messages.*
> > > > > > > > >> >
> > > > > > > > >> > - We agree that time_mode could be introduced to other
> > > > > > > window-based
> > > > > > > > >> > operators down the road. We also want to support early
> > > fire
> > > > > for
> > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the code
> > > change
> > > > on
> > > > > > it
> > > > > > > > >> > <
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > >> >
> > > > > > > > >> > in the proposal.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >Thanks again for your time and feedback! I’ve updated the
> > > > > proposal
> > > > > > > with
> > > > > > > > >> >these points. Please let me know if there’s anything
> else I
> > > > > should
> > > > > > > > >> address.
> > > > > > > > >> >
> > > > > > > > >> >Best,
> > > > > > > > >> >Weiqing
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> xyzhong131@163.com>
> > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a
> > few
> > > > > > > > questions:
> > > > > > > > >> >>
> > > > > > > > >> >> 1. Currently, there are some configs related to early
> > > firing
> > > > > > > > available
> > > > > > > > >> to
> > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > >> >>
> > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > > > documentation
> > > > > > > > states
> > > > > > > > >> >> that they are only applicable to the Window operator,
> > > > > > > > >> >>
> > > > > > > > >> >> it seems possible that they may also be relevant in the
> > > > context
> > > > > > of
> > > > > > > > this
> > > > > > > > >> >> FLIP. Otherwise, having different early firing
> behaviors
> > > > > > > > >> >>
> > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > >> >>
> > > > > > > > >> >> 2. The design of `time_mode` is excellent. Similar to
> > point
> > > > 1,
> > > > > > > > perhaps
> > > > > > > > >> we
> > > > > > > > >> >> can introduce it to other window-related operators
> > > > > > > > >> >>
> > > > > > > > >> >> in the future.
> > > > > > > > >> >>
> > > > > > > > >> >> 3. You need to modify the
> > > FlinkChangelogModeInferenceProgram
> > > > to
> > > > > > > > ensure
> > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > >> >>
> > > > > > > > >> >> early firing enabled are aware of retract or upsert
> > > messages.
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> --
> > > > > > > > >> >>
> > > > > > > > >> >> Best！
> > > > > > > > >> >> Xuyang
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > xingcanc@gmail.com>
> > > > > > wrote:
> > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > >> >> >
> > > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire for
> time
> > > > > > interval
> > > > > > > > outer
> > > > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > > > >> >> >
> > > > > > > > >> >> >1. The current output stream of a time interval outer
> > join
> > > > is
> > > > > an
> > > > > > > > >> >> >append-only stream. This change will make it a
> potential
> > > > > > > retractable
> > > > > > > > >> >> >stream. I'm not sure if the planner supports a dynamic
> > > > output
> > > > > > type
> > > > > > > > like
> > > > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > > > >> >> >2. What's the use case when the downstream components
> > need
> > > > to
> > > > > > get
> > > > > > > > the
> > > > > > > > >> >> early
> > > > > > > > >> >> >fired results regularly?
> > > > > > > > >> >> >3. The time interval join operator itself is not quite
> > > > > efficient
> > > > > > > > when
> > > > > > > > >> the
> > > > > > > > >> >> >state becomes large. Will there be any extra overhead
> > > after
> > > > > > > > introducing
> > > > > > > > >> >> >this feature?
> > > > > > > > >> >> >
> > > > > > > > >> >> >Thanks,
> > > > > > > > >> >> >Xingcan
> > > > > > > > >> >> >
> > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > yangweiqing001@gmail.com>
> > > > > > > > >> >> >wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >> Hi all,
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Just a gentle reminder regarding the proposal I
> shared
> > > on
> > > > > > early
> > > > > > > > fire
> > > > > > > > >> >> >> support for Flink SQL interval joins. I’d greatly
> > > > appreciate
> > > > > > > your
> > > > > > > > >> >> feedback
> > > > > > > > >> >> >> or suggestions.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > > >> >> >> <
> > > > > > > > >> >> >>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Thank you!
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Best,
> > > > > > > > >> >> >> Weiqing
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > > > >> yangweiqing001@gmail.com
> > > > > > > > >> >> >
> > > > > > > > >> >> >> wrote:
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> > Hi all,
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> > I’d like to initiate a discussion about
> introducing
> > > > early
> > > > > > fire
> > > > > > > > >> support
> > > > > > > > >> >> >> for
> > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> > *Motivation*
> > > > > > > > >> >> >> > In many streaming applications, particularly
> > real-time
> > > > > > > analytics
> > > > > > > > >> and
> > > > > > > > >> >> >> > monitoring systems, it is valuable to obtain
> partial
> > > > > results
> > > > > > > > >> earlier
> > > > > > > > >> >> >> rather
> > > > > > > > >> >> >> > than waiting for full join conditions to finalize.
> > For
> > > > > Flink
> > > > > > > SQL
> > > > > > > > >> >> interval
> > > > > > > > >> >> >> > joins, results are typically delayed until
> > watermarks
> > > > > ensure
> > > > > > > no
> > > > > > > > >> more
> > > > > > > > >> >> >> > matches can occur. This delay can be challenging
> for
> > > > > > scenarios
> > > > > > > > that
> > > > > > > > >> >> >> require
> > > > > > > > >> >> >> > fast feedback. Early fire support addresses this
> by
> > > > > emitting
> > > > > > > > >> >> intermediate
> > > > > > > > >> >> >> > results speculatively and using retractions or
> > updates
> > > > to
> > > > > > > > maintain
> > > > > > > > >> >> >> eventual
> > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > >> >> >> > <
> > > > > > > > >> >> >>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> > Your feedback and ideas are welcome to refine this
> > > > > feature.
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >> > Thanks,
> > > > > > > > >> >> >> > Weiqing
> > > > > > > > >> >> >> >
> > > > > > > > >> >> >>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Xingcan Cui <xi...@gmail.com>.
Hi Weiqing,
I don't have any more questions. The doc looks good to me.
Thanks,
Xingcan
On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <vs...@asu.edu>
wrote:
> Hi Weiqing,
>
> Thanks, that makes sense! Looks like I missed it.
>
> Regards
> Venkata krishnan
>
>
> On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Hi Venkata,
> >
> > * > where only one earlyFire is fired The DELAY *
> >
> > The DELAY option mentioned in the Public Interfaces section
> > <
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > >
> > of the proposal can produce a single early fire per interval, aligning
> with
> > your suggestion. How it integrates with existing early-fire
> configurations
> > are mentioned here
> > <
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > >.
> > Let me know if you have any further questions or feedback!
> >
> > Thanks,
> > Weiqing
> >
> > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > vsowrira@asu.edu> wrote:
> >
> > > Thanks for the response!
> > >
> > > > If the optional `interval` in the proposal is enabled, early-fire
> > outputs
> > > will occur repeatedly at every earlyFireInterval, not just once. After
> > that
> > > interval concludes, there will also be a final emission.
> > >
> > > One more question, would it make sense to support a mechanism through
> > > configuration where only one earlyFire is fired for an interval or
> > window?
> > > This would also simplify the state overhead for use cases where only
> one
> > > early fire is required within a window or interval. Thoughts?
> > >
> > > Regards
> > > Venkata krishnan
> > >
> > >
> > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <yangweiqing001@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Venkata,
> > > >
> > > > Thanks for your feedback!
> > > >
> > > > If the optional `interval` in the proposal is enabled, early-fire
> > outputs
> > > > will occur repeatedly at every earlyFireInterval, not just once.
> After
> > > that
> > > > interval concludes, there will also be a final emission.
> > > >
> > > > We haven’t included a late-fire mechanism for interval joins in this
> > > FLIP,
> > > > but it’s certainly something we can consider for future efforts!
> > > >
> > > > Best regards,
> > > > Weiqing
> > > >
> > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > vsowrira@asu.edu>
> > > > wrote:
> > > >
> > > > > Weiqing,
> > > > >
> > > > > Thanks, this is a great addition to Flink SQL. Also instead of
> > > > controlling
> > > > > and configuring through Flink configs unlike the older window
> > > > aggregation,
> > > > > hints seems to be a much better approach. This enables a
> customizable
> > > > early
> > > > > fire behavior for individual interval joins.
> > > > >
> > > > > Couple of questions:
> > > > >
> > > > > 1. Does the *early fire* emit an output every earlyFireInterval
> time
> > or
> > > > > will it be a one time output emission and another output emitted at
> > the
> > > > end
> > > > > of the interval?
> > > > > 2. Are there plans to support *late fire *similar to the window
> > > > > aggregations in later FLIPs?
> > > > >
> > > > > Regards
> > > > > Venkata krishnan
> > > > >
> > > > >
> > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > yangweiqing001@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for reviewing, Xuyang!
> > > > > >
> > > > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > > > >
> > > > > > If no further objections arise from anyone, I’ll proceed to mark
> > FLIP
> > > > as
> > > > > > ready for voting.
> > > > > >
> > > > > > Best regards,
> > > > > > Weiqing
> > > > > >
> > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com>
> wrote:
> > > > > >
> > > > > > > LGTM overall. Thanks for updating. I have no problem and +1 for
> > > this
> > > > > > > feature.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best！
> > > > > > > Xuyang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <yangweiqing001@gmail.com
> >
> > > 写道：
> > > > > > > >Hi Xuyang,
> > > > > > > >
> > > > > > > >Thank you for your detailed feedback! I’ve updated the
> proposal
> > > doc
> > > > > > > >accordingly. Please feel free to take another look and let me
> > know
> > > > if
> > > > > > you
> > > > > > > >have any further thoughts or suggestions.
> > > > > > > >
> > > > > > > >Best regards,
> > > > > > > >Weiqing
> > > > > > > >
> > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com>
> > > wrote:
> > > > > > > >
> > > > > > > >> Hi, Weiqing.
> > > > > > > >>
> > > > > > > >> After reading the new FLIP, I have no issues with the part
> > > `public
> > > > > > > >> interface`. I only have some questions regarding
> > > > > > > >>
> > > > > > > >> the details in the Proposed Changes section.
> > > > > > > >>
> > > > > > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin
> > > node,
> > > > > > IIUC:
> > > > > > > >>
> > > > > > > >> - When early firing is enabled, the UpdateKind of the
> > > IntervalJoin
> > > > > can
> > > > > > > be
> > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > >>
> > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > > requirements
> > > > > of
> > > > > > > the
> > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > >>
> > > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > > IntervalJoin
> > > > > is
> > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > >>
> > > > > > > >> - Nevertheless, whether early firing is enabled or disabled,
> > the
> > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > >>
> > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> Best！
> > > > > > > >> Xuyang
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > yangweiqing001@gmail.com>
> > > > > > > wrote:
> > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > >> >
> > > > > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > > > > >> >
> > > > > > > >> >*> 1. The current output stream of a time interval outer
> join
> > > is
> > > > an
> > > > > > > >> >append-only stream. This change will make it a potential
> > > > > retractable
> > > > > > > >> >stream. I'm not sure if the planner supports a dynamic
> output
> > > > type
> > > > > > like
> > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > - Yes, enabling early firing on time interval outer
> joins
> > > can
> > > > > emit
> > > > > > > >> > retractions when previously emitted rows are updated or
> > > > > > invalidated
> > > > > > > by
> > > > > > > >> > later matches. I’ve updated the proposal (Planner
> > Awareness
> > > > > > > >> > <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > >> >
> > > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > > >> > <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > >> >)
> > > > > > > >> > to clarify that the stream might switch from append-only
> > to
> > > a
> > > > > > > >> > retract/upsert stream. Let me know if anything is
> missing.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >*> 2. What's the use case when the downstream components
> need
> > > to
> > > > > get
> > > > > > > the
> > > > > > > >> >early fired results regularly?*
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > - The new INTERVAL option (in addition to DELAY) allows
> > > > periodic
> > > > > > > >> updates
> > > > > > > >> > (e.g., every 10 minutes) after the initial delay. This
> > > > captures
> > > > > > how
> > > > > > > >> results
> > > > > > > >> > evolve over time, similar to Apache Beam’s “Repeatedly”
> > > > option.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >*> 3. The time interval join operator itself is not quite
> > > > efficient
> > > > > > > when
> > > > > > > >> >the state becomes large. Will there be any extra overhead
> > after
> > > > > > > >> introducing
> > > > > > > >> >this feature?*
> > > > > > > >> >
> > > > > > > >> > - Early fire does introduce some overhead by potentially
> > > > > emitting
> > > > > > > >> > partial matches multiple times with retraction (avoiding
> > > > > duplicate
> > > > > > > >> outputs
> > > > > > > >> > though). However, if it’s disabled, there is no
> additional
> > > > cost.
> > > > > > > Most
> > > > > > > >> users
> > > > > > > >> > find the performance trade-off acceptable for the
> > real-time
> > > > > > > insights it
> > > > > > > >> > provides.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >*> 1. Currently, there are some configs related to early
> > firing
> > > > > > > available
> > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > >> >**lay`.
> > > > > > > >> >Although their documentation states that they are only
> > > applicable
> > > > > to
> > > > > > > the
> > > > > > > >> >Window operator, it seems possible that they may also be
> > > relevant
> > > > > in
> > > > > > > the
> > > > > > > >> >context of this FLIP. Otherwise, having different early
> > firing
> > > > > > > behaviors
> > > > > > > >> >for different operators could confuse users.*
> > > > > > > >> >
> > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> confusion.
> > > I’ve
> > > > > > > added a
> > > > > > > >> > section
> > > > > > > >> > <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > >> >
> > > > > > > >> >in
> > > > > > > >> > the proposal highlighting that we should align
> hint-based
> > > > > interval
> > > > > > > join
> > > > > > > >> > configurations with the existing table.exec.emit.*
> > settings.
> > > > > > > >> Suggestions
> > > > > > > >> > on how to make the unification are welcome! We plan to
> > > extend
> > > > > > early
> > > > > > > >> firing
> > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar to
> > point
> > > 1,
> > > > > > > perhaps
> > > > > > > >> >we can introduce it to other window-related operators in
> the
> > > > > future.>
> > > > > > > 3.
> > > > > > > >> >You need to modify the FlinkChangelogModeInferenceProgram
> to
> > > > ensure
> > > > > > > that
> > > > > > > >> >downstream nodes of interval joins with early firing
> enabled
> > > are
> > > > > > aware
> > > > > > > of
> > > > > > > >> >retract or upsert messages.*
> > > > > > > >> >
> > > > > > > >> > - We agree that time_mode could be introduced to other
> > > > > > window-based
> > > > > > > >> > operators down the road. We also want to support early
> > fire
> > > > for
> > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the code
> > change
> > > on
> > > > > it
> > > > > > > >> > <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > >> >
> > > > > > > >> > in the proposal.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >Thanks again for your time and feedback! I’ve updated the
> > > > proposal
> > > > > > with
> > > > > > > >> >these points. Please let me know if there’s anything else I
> > > > should
> > > > > > > >> address.
> > > > > > > >> >
> > > > > > > >> >Best,
> > > > > > > >> >Weiqing
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com>
> > > > wrote:
> > > > > > > >> >
> > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a
> few
> > > > > > > questions:
> > > > > > > >> >>
> > > > > > > >> >> 1. Currently, there are some configs related to early
> > firing
> > > > > > > available
> > > > > > > >> to
> > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > >> >>
> > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > > documentation
> > > > > > > states
> > > > > > > >> >> that they are only applicable to the Window operator,
> > > > > > > >> >>
> > > > > > > >> >> it seems possible that they may also be relevant in the
> > > context
> > > > > of
> > > > > > > this
> > > > > > > >> >> FLIP. Otherwise, having different early firing behaviors
> > > > > > > >> >>
> > > > > > > >> >> for different operators could confuse users.
> > > > > > > >> >>
> > > > > > > >> >> 2. The design of `time_mode` is excellent. Similar to
> point
> > > 1,
> > > > > > > perhaps
> > > > > > > >> we
> > > > > > > >> >> can introduce it to other window-related operators
> > > > > > > >> >>
> > > > > > > >> >> in the future.
> > > > > > > >> >>
> > > > > > > >> >> 3. You need to modify the
> > FlinkChangelogModeInferenceProgram
> > > to
> > > > > > > ensure
> > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > >> >>
> > > > > > > >> >> early firing enabled are aware of retract or upsert
> > messages.
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >> --
> > > > > > > >> >>
> > > > > > > >> >> Best！
> > > > > > > >> >> Xuyang
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> xingcanc@gmail.com>
> > > > > wrote:
> > > > > > > >> >> >Hi Weiqing,
> > > > > > > >> >> >
> > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire for time
> > > > > interval
> > > > > > > outer
> > > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > > >> >> >
> > > > > > > >> >> >1. The current output stream of a time interval outer
> join
> > > is
> > > > an
> > > > > > > >> >> >append-only stream. This change will make it a potential
> > > > > > retractable
> > > > > > > >> >> >stream. I'm not sure if the planner supports a dynamic
> > > output
> > > > > type
> > > > > > > like
> > > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > > >> >> >2. What's the use case when the downstream components
> need
> > > to
> > > > > get
> > > > > > > the
> > > > > > > >> >> early
> > > > > > > >> >> >fired results regularly?
> > > > > > > >> >> >3. The time interval join operator itself is not quite
> > > > efficient
> > > > > > > when
> > > > > > > >> the
> > > > > > > >> >> >state becomes large. Will there be any extra overhead
> > after
> > > > > > > introducing
> > > > > > > >> >> >this feature?
> > > > > > > >> >> >
> > > > > > > >> >> >Thanks,
> > > > > > > >> >> >Xingcan
> > > > > > > >> >> >
> > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > yangweiqing001@gmail.com>
> > > > > > > >> >> >wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> Hi all,
> > > > > > > >> >> >>
> > > > > > > >> >> >> Just a gentle reminder regarding the proposal I shared
> > on
> > > > > early
> > > > > > > fire
> > > > > > > >> >> >> support for Flink SQL interval joins. I’d greatly
> > > appreciate
> > > > > > your
> > > > > > > >> >> feedback
> > > > > > > >> >> >> or suggestions.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > >> >> >> <
> > > > > > > >> >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > >> >> >> >
> > > > > > > >> >> >>
> > > > > > > >> >> >> Thank you!
> > > > > > > >> >> >>
> > > > > > > >> >> >> Best,
> > > > > > > >> >> >> Weiqing
> > > > > > > >> >> >>
> > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > > >> yangweiqing001@gmail.com
> > > > > > > >> >> >
> > > > > > > >> >> >> wrote:
> > > > > > > >> >> >>
> > > > > > > >> >> >> > Hi all,
> > > > > > > >> >> >> >
> > > > > > > >> >> >> > I’d like to initiate a discussion about introducing
> > > early
> > > > > fire
> > > > > > > >> support
> > > > > > > >> >> >> for
> > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > >> >> >> >
> > > > > > > >> >> >> > *Motivation*
> > > > > > > >> >> >> > In many streaming applications, particularly
> real-time
> > > > > > analytics
> > > > > > > >> and
> > > > > > > >> >> >> > monitoring systems, it is valuable to obtain partial
> > > > results
> > > > > > > >> earlier
> > > > > > > >> >> >> rather
> > > > > > > >> >> >> > than waiting for full join conditions to finalize.
> For
> > > > Flink
> > > > > > SQL
> > > > > > > >> >> interval
> > > > > > > >> >> >> > joins, results are typically delayed until
> watermarks
> > > > ensure
> > > > > > no
> > > > > > > >> more
> > > > > > > >> >> >> > matches can occur. This delay can be challenging for
> > > > > scenarios
> > > > > > > that
> > > > > > > >> >> >> require
> > > > > > > >> >> >> > fast feedback. Early fire support addresses this by
> > > > emitting
> > > > > > > >> >> intermediate
> > > > > > > >> >> >> > results speculatively and using retractions or
> updates
> > > to
> > > > > > > maintain
> > > > > > > >> >> >> eventual
> > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > >> >> >> >
> > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > >> >> >> > <
> > > > > > > >> >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > >> >> >> >
> > > > > > > >> >> >> >
> > > > > > > >> >> >> > Your feedback and ideas are welcome to refine this
> > > > feature.
> > > > > > > >> >> >> >
> > > > > > > >> >> >> > Thanks,
> > > > > > > >> >> >> > Weiqing
> > > > > > > >> >> >> >
> > > > > > > >> >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Hi Weiqing,
Thanks, that makes sense! Looks like I missed it.
Regards
Venkata krishnan
On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi Venkata,
>
> * > where only one earlyFire is fired The DELAY *
>
> The DELAY option mentioned in the Public Interfaces section
> <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> >
> of the proposal can produce a single early fire per interval, aligning with
> your suggestion. How it integrates with existing early-fire configurations
> are mentioned here
> <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> >.
> Let me know if you have any further questions or feedback!
>
> Thanks,
> Weiqing
>
> On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> vsowrira@asu.edu> wrote:
>
> > Thanks for the response!
> >
> > > If the optional `interval` in the proposal is enabled, early-fire
> outputs
> > will occur repeatedly at every earlyFireInterval, not just once. After
> that
> > interval concludes, there will also be a final emission.
> >
> > One more question, would it make sense to support a mechanism through
> > configuration where only one earlyFire is fired for an interval or
> window?
> > This would also simplify the state overhead for use cases where only one
> > early fire is required within a window or interval. Thoughts?
> >
> > Regards
> > Venkata krishnan
> >
> >
> > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <ya...@gmail.com>
> > wrote:
> >
> > > Hi Venkata,
> > >
> > > Thanks for your feedback!
> > >
> > > If the optional `interval` in the proposal is enabled, early-fire
> outputs
> > > will occur repeatedly at every earlyFireInterval, not just once. After
> > that
> > > interval concludes, there will also be a final emission.
> > >
> > > We haven’t included a late-fire mechanism for interval joins in this
> > FLIP,
> > > but it’s certainly something we can consider for future efforts!
> > >
> > > Best regards,
> > > Weiqing
> > >
> > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > vsowrira@asu.edu>
> > > wrote:
> > >
> > > > Weiqing,
> > > >
> > > > Thanks, this is a great addition to Flink SQL. Also instead of
> > > controlling
> > > > and configuring through Flink configs unlike the older window
> > > aggregation,
> > > > hints seems to be a much better approach. This enables a customizable
> > > early
> > > > fire behavior for individual interval joins.
> > > >
> > > > Couple of questions:
> > > >
> > > > 1. Does the *early fire* emit an output every earlyFireInterval time
> or
> > > > will it be a one time output emission and another output emitted at
> the
> > > end
> > > > of the interval?
> > > > 2. Are there plans to support *late fire *similar to the window
> > > > aggregations in later FLIPs?
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> yangweiqing001@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks for reviewing, Xuyang!
> > > > >
> > > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > > >
> > > > > If no further objections arise from anyone, I’ll proceed to mark
> FLIP
> > > as
> > > > > ready for voting.
> > > > >
> > > > > Best regards,
> > > > > Weiqing
> > > > >
> > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
> > > > >
> > > > > > LGTM overall. Thanks for updating. I have no problem and +1 for
> > this
> > > > > > feature.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best！
> > > > > > Xuyang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com>
> > 写道：
> > > > > > >Hi Xuyang,
> > > > > > >
> > > > > > >Thank you for your detailed feedback! I’ve updated the proposal
> > doc
> > > > > > >accordingly. Please feel free to take another look and let me
> know
> > > if
> > > > > you
> > > > > > >have any further thoughts or suggestions.
> > > > > > >
> > > > > > >Best regards,
> > > > > > >Weiqing
> > > > > > >
> > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com>
> > wrote:
> > > > > > >
> > > > > > >> Hi, Weiqing.
> > > > > > >>
> > > > > > >> After reading the new FLIP, I have no issues with the part
> > `public
> > > > > > >> interface`. I only have some questions regarding
> > > > > > >>
> > > > > > >> the details in the Proposed Changes section.
> > > > > > >>
> > > > > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin
> > node,
> > > > > IIUC:
> > > > > > >>
> > > > > > >> - When early firing is enabled, the UpdateKind of the
> > IntervalJoin
> > > > can
> > > > > > be
> > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > >>
> > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > requirements
> > > > of
> > > > > > the
> > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > >>
> > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > IntervalJoin
> > > > is
> > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > >>
> > > > > > >> - Nevertheless, whether early firing is enabled or disabled,
> the
> > > > > > >> IntervalJoin should always require its input to keep
> > > > > > >>
> > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >>
> > > > > > >> Best！
> > > > > > >> Xuyang
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > yangweiqing001@gmail.com>
> > > > > > wrote:
> > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > >> >
> > > > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > > > >> >
> > > > > > >> >*> 1. The current output stream of a time interval outer join
> > is
> > > an
> > > > > > >> >append-only stream. This change will make it a potential
> > > > retractable
> > > > > > >> >stream. I'm not sure if the planner supports a dynamic output
> > > type
> > > > > like
> > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > - Yes, enabling early firing on time interval outer joins
> > can
> > > > emit
> > > > > > >> > retractions when previously emitted rows are updated or
> > > > > invalidated
> > > > > > by
> > > > > > >> > later matches. I’ve updated the proposal (Planner
> Awareness
> > > > > > >> > <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > >> >
> > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > >> > <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > >> >)
> > > > > > >> > to clarify that the stream might switch from append-only
> to
> > a
> > > > > > >> > retract/upsert stream. Let me know if anything is missing.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >*> 2. What's the use case when the downstream components need
> > to
> > > > get
> > > > > > the
> > > > > > >> >early fired results regularly?*
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > - The new INTERVAL option (in addition to DELAY) allows
> > > periodic
> > > > > > >> updates
> > > > > > >> > (e.g., every 10 minutes) after the initial delay. This
> > > captures
> > > > > how
> > > > > > >> results
> > > > > > >> > evolve over time, similar to Apache Beam’s “Repeatedly”
> > > option.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >*> 3. The time interval join operator itself is not quite
> > > efficient
> > > > > > when
> > > > > > >> >the state becomes large. Will there be any extra overhead
> after
> > > > > > >> introducing
> > > > > > >> >this feature?*
> > > > > > >> >
> > > > > > >> > - Early fire does introduce some overhead by potentially
> > > > emitting
> > > > > > >> > partial matches multiple times with retraction (avoiding
> > > > duplicate
> > > > > > >> outputs
> > > > > > >> > though). However, if it’s disabled, there is no additional
> > > cost.
> > > > > > Most
> > > > > > >> users
> > > > > > >> > find the performance trade-off acceptable for the
> real-time
> > > > > > insights it
> > > > > > >> > provides.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >*> 1. Currently, there are some configs related to early
> firing
> > > > > > available
> > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > >> >`table.exec.emit.early-fire.de <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > >> >**lay`.
> > > > > > >> >Although their documentation states that they are only
> > applicable
> > > > to
> > > > > > the
> > > > > > >> >Window operator, it seems possible that they may also be
> > relevant
> > > > in
> > > > > > the
> > > > > > >> >context of this FLIP. Otherwise, having different early
> firing
> > > > > > behaviors
> > > > > > >> >for different operators could confuse users.*
> > > > > > >> >
> > > > > > >> > - +1 on unifying early-fire behaviors to avoid confusion.
> > I’ve
> > > > > > added a
> > > > > > >> > section
> > > > > > >> > <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > >> >
> > > > > > >> >in
> > > > > > >> > the proposal highlighting that we should align hint-based
> > > > interval
> > > > > > join
> > > > > > >> > configurations with the existing table.exec.emit.*
> settings.
> > > > > > >> Suggestions
> > > > > > >> > on how to make the unification are welcome! We plan to
> > extend
> > > > > early
> > > > > > >> firing
> > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar to
> point
> > 1,
> > > > > > perhaps
> > > > > > >> >we can introduce it to other window-related operators in the
> > > > future.>
> > > > > > 3.
> > > > > > >> >You need to modify the FlinkChangelogModeInferenceProgram to
> > > ensure
> > > > > > that
> > > > > > >> >downstream nodes of interval joins with early firing enabled
> > are
> > > > > aware
> > > > > > of
> > > > > > >> >retract or upsert messages.*
> > > > > > >> >
> > > > > > >> > - We agree that time_mode could be introduced to other
> > > > > window-based
> > > > > > >> > operators down the road. We also want to support early
> fire
> > > for
> > > > > > >> > window join. Also, thanks for highlighting
> > > > > > >> > FlinkChangelogModeInferenceProgram! I added the code
> change
> > on
> > > > it
> > > > > > >> > <
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > >> >
> > > > > > >> > in the proposal.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >Thanks again for your time and feedback! I’ve updated the
> > > proposal
> > > > > with
> > > > > > >> >these points. Please let me know if there’s anything else I
> > > should
> > > > > > >> address.
> > > > > > >> >
> > > > > > >> >Best,
> > > > > > >> >Weiqing
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com>
> > > wrote:
> > > > > > >> >
> > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> > > > > > questions:
> > > > > > >> >>
> > > > > > >> >> 1. Currently, there are some configs related to early
> firing
> > > > > > available
> > > > > > >> to
> > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > >> >>
> > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > documentation
> > > > > > states
> > > > > > >> >> that they are only applicable to the Window operator,
> > > > > > >> >>
> > > > > > >> >> it seems possible that they may also be relevant in the
> > context
> > > > of
> > > > > > this
> > > > > > >> >> FLIP. Otherwise, having different early firing behaviors
> > > > > > >> >>
> > > > > > >> >> for different operators could confuse users.
> > > > > > >> >>
> > > > > > >> >> 2. The design of `time_mode` is excellent. Similar to point
> > 1,
> > > > > > perhaps
> > > > > > >> we
> > > > > > >> >> can introduce it to other window-related operators
> > > > > > >> >>
> > > > > > >> >> in the future.
> > > > > > >> >>
> > > > > > >> >> 3. You need to modify the
> FlinkChangelogModeInferenceProgram
> > to
> > > > > > ensure
> > > > > > >> >> that downstream nodes of interval joins with
> > > > > > >> >>
> > > > > > >> >> early firing enabled are aware of retract or upsert
> messages.
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >>
> > > > > > >> >> Best！
> > > > > > >> >> Xuyang
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com>
> > > > wrote:
> > > > > > >> >> >Hi Weiqing,
> > > > > > >> >> >
> > > > > > >> >> >Thanks for the proposal. IMO, adding early fire for time
> > > > interval
> > > > > > outer
> > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > >> >> >
> > > > > > >> >> >1. The current output stream of a time interval outer join
> > is
> > > an
> > > > > > >> >> >append-only stream. This change will make it a potential
> > > > > retractable
> > > > > > >> >> >stream. I'm not sure if the planner supports a dynamic
> > output
> > > > type
> > > > > > like
> > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > >> >> >2. What's the use case when the downstream components need
> > to
> > > > get
> > > > > > the
> > > > > > >> >> early
> > > > > > >> >> >fired results regularly?
> > > > > > >> >> >3. The time interval join operator itself is not quite
> > > efficient
> > > > > > when
> > > > > > >> the
> > > > > > >> >> >state becomes large. Will there be any extra overhead
> after
> > > > > > introducing
> > > > > > >> >> >this feature?
> > > > > > >> >> >
> > > > > > >> >> >Thanks,
> > > > > > >> >> >Xingcan
> > > > > > >> >> >
> > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > yangweiqing001@gmail.com>
> > > > > > >> >> >wrote:
> > > > > > >> >> >
> > > > > > >> >> >> Hi all,
> > > > > > >> >> >>
> > > > > > >> >> >> Just a gentle reminder regarding the proposal I shared
> on
> > > > early
> > > > > > fire
> > > > > > >> >> >> support for Flink SQL interval joins. I’d greatly
> > appreciate
> > > > > your
> > > > > > >> >> feedback
> > > > > > >> >> >> or suggestions.
> > > > > > >> >> >>
> > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > >> >> >> <
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >> >> Thank you!
> > > > > > >> >> >>
> > > > > > >> >> >> Best,
> > > > > > >> >> >> Weiqing
> > > > > > >> >> >>
> > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > >> yangweiqing001@gmail.com
> > > > > > >> >> >
> > > > > > >> >> >> wrote:
> > > > > > >> >> >>
> > > > > > >> >> >> > Hi all,
> > > > > > >> >> >> >
> > > > > > >> >> >> > I’d like to initiate a discussion about introducing
> > early
> > > > fire
> > > > > > >> support
> > > > > > >> >> >> for
> > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > >> >> >> >
> > > > > > >> >> >> > *Motivation*
> > > > > > >> >> >> > In many streaming applications, particularly real-time
> > > > > analytics
> > > > > > >> and
> > > > > > >> >> >> > monitoring systems, it is valuable to obtain partial
> > > results
> > > > > > >> earlier
> > > > > > >> >> >> rather
> > > > > > >> >> >> > than waiting for full join conditions to finalize. For
> > > Flink
> > > > > SQL
> > > > > > >> >> interval
> > > > > > >> >> >> > joins, results are typically delayed until watermarks
> > > ensure
> > > > > no
> > > > > > >> more
> > > > > > >> >> >> > matches can occur. This delay can be challenging for
> > > > scenarios
> > > > > > that
> > > > > > >> >> >> require
> > > > > > >> >> >> > fast feedback. Early fire support addresses this by
> > > emitting
> > > > > > >> >> intermediate
> > > > > > >> >> >> > results speculatively and using retractions or updates
> > to
> > > > > > maintain
> > > > > > >> >> >> eventual
> > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > >> >> >> >
> > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > >> >> >> > <
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > >> >> >> >
> > > > > > >> >> >> >
> > > > > > >> >> >> > Your feedback and ideas are welcome to refine this
> > > feature.
> > > > > > >> >> >> >
> > > > > > >> >> >> > Thanks,
> > > > > > >> >> >> > Weiqing
> > > > > > >> >> >> >
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Venkata,
* > where only one earlyFire is fired The DELAY *
The DELAY option mentioned in the Public Interfaces section
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.yp9ng89zwc1z>
of the proposal can produce a single early fire per interval, aligning with
your suggestion. How it integrates with existing early-fire configurations
are mentioned here
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q>.
Let me know if you have any further questions or feedback!
Thanks,
Weiqing
On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
vsowrira@asu.edu> wrote:
> Thanks for the response!
>
> > If the optional `interval` in the proposal is enabled, early-fire outputs
> will occur repeatedly at every earlyFireInterval, not just once. After that
> interval concludes, there will also be a final emission.
>
> One more question, would it make sense to support a mechanism through
> configuration where only one earlyFire is fired for an interval or window?
> This would also simplify the state overhead for use cases where only one
> early fire is required within a window or interval. Thoughts?
>
> Regards
> Venkata krishnan
>
>
> On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Hi Venkata,
> >
> > Thanks for your feedback!
> >
> > If the optional `interval` in the proposal is enabled, early-fire outputs
> > will occur repeatedly at every earlyFireInterval, not just once. After
> that
> > interval concludes, there will also be a final emission.
> >
> > We haven’t included a late-fire mechanism for interval joins in this
> FLIP,
> > but it’s certainly something we can consider for future efforts!
> >
> > Best regards,
> > Weiqing
> >
> > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > vsowrira@asu.edu>
> > wrote:
> >
> > > Weiqing,
> > >
> > > Thanks, this is a great addition to Flink SQL. Also instead of
> > controlling
> > > and configuring through Flink configs unlike the older window
> > aggregation,
> > > hints seems to be a much better approach. This enables a customizable
> > early
> > > fire behavior for individual interval joins.
> > >
> > > Couple of questions:
> > >
> > > 1. Does the *early fire* emit an output every earlyFireInterval time or
> > > will it be a one time output emission and another output emitted at the
> > end
> > > of the interval?
> > > 2. Are there plans to support *late fire *similar to the window
> > > aggregations in later FLIPs?
> > >
> > > Regards
> > > Venkata krishnan
> > >
> > >
> > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <yangweiqing001@gmail.com
> >
> > > wrote:
> > >
> > > > Thanks for reviewing, Xuyang!
> > > >
> > > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > > >
> > > > If no further objections arise from anyone, I’ll proceed to mark FLIP
> > as
> > > > ready for voting.
> > > >
> > > > Best regards,
> > > > Weiqing
> > > >
> > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
> > > >
> > > > > LGTM overall. Thanks for updating. I have no problem and +1 for
> this
> > > > > feature.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best！
> > > > > Xuyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com>
> 写道：
> > > > > >Hi Xuyang,
> > > > > >
> > > > > >Thank you for your detailed feedback! I’ve updated the proposal
> doc
> > > > > >accordingly. Please feel free to take another look and let me know
> > if
> > > > you
> > > > > >have any further thoughts or suggestions.
> > > > > >
> > > > > >Best regards,
> > > > > >Weiqing
> > > > > >
> > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com>
> wrote:
> > > > > >
> > > > > >> Hi, Weiqing.
> > > > > >>
> > > > > >> After reading the new FLIP, I have no issues with the part
> `public
> > > > > >> interface`. I only have some questions regarding
> > > > > >>
> > > > > >> the details in the Proposed Changes section.
> > > > > >>
> > > > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin
> node,
> > > > IIUC:
> > > > > >>
> > > > > >> - When early firing is enabled, the UpdateKind of the
> IntervalJoin
> > > can
> > > > > be
> > > > > >> either ONLY_UPDATE_AFTER or
> > > > > >>
> > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > requirements
> > > of
> > > > > the
> > > > > >> sink. And the ModifyKind is always ALL.
> > > > > >>
> > > > > >> - When early firing is disabled, the UpdateKind of the
> > IntervalJoin
> > > is
> > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > >>
> > > > > >> - Nevertheless, whether early firing is enabled or disabled, the
> > > > > >> IntervalJoin should always require its input to keep
> > > > > >>
> > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> Best！
> > > > > >> Xuyang
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> yangweiqing001@gmail.com>
> > > > > wrote:
> > > > > >> >Hi Xingcan and Xuyang,
> > > > > >> >
> > > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > > >> >
> > > > > >> >*> 1. The current output stream of a time interval outer join
> is
> > an
> > > > > >> >append-only stream. This change will make it a potential
> > > retractable
> > > > > >> >stream. I'm not sure if the planner supports a dynamic output
> > type
> > > > like
> > > > > >> >that. Could you add this part to your design doc?*
> > > > > >> >
> > > > > >> >
> > > > > >> > - Yes, enabling early firing on time interval outer joins
> can
> > > emit
> > > > > >> > retractions when previously emitted rows are updated or
> > > > invalidated
> > > > > by
> > > > > >> > later matches. I’ve updated the proposal (Planner Awareness
> > > > > >> > <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > >> >
> > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > >> > <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > >> >)
> > > > > >> > to clarify that the stream might switch from append-only to
> a
> > > > > >> > retract/upsert stream. Let me know if anything is missing.
> > > > > >> >
> > > > > >> >
> > > > > >> >*> 2. What's the use case when the downstream components need
> to
> > > get
> > > > > the
> > > > > >> >early fired results regularly?*
> > > > > >> >
> > > > > >> >
> > > > > >> > - The new INTERVAL option (in addition to DELAY) allows
> > periodic
> > > > > >> updates
> > > > > >> > (e.g., every 10 minutes) after the initial delay. This
> > captures
> > > > how
> > > > > >> results
> > > > > >> > evolve over time, similar to Apache Beam’s “Repeatedly”
> > option.
> > > > > >> >
> > > > > >> >
> > > > > >> >*> 3. The time interval join operator itself is not quite
> > efficient
> > > > > when
> > > > > >> >the state becomes large. Will there be any extra overhead after
> > > > > >> introducing
> > > > > >> >this feature?*
> > > > > >> >
> > > > > >> > - Early fire does introduce some overhead by potentially
> > > emitting
> > > > > >> > partial matches multiple times with retraction (avoiding
> > > duplicate
> > > > > >> outputs
> > > > > >> > though). However, if it’s disabled, there is no additional
> > cost.
> > > > > Most
> > > > > >> users
> > > > > >> > find the performance trade-off acceptable for the real-time
> > > > > insights it
> > > > > >> > provides.
> > > > > >> >
> > > > > >> >
> > > > > >> >*> 1. Currently, there are some configs related to early firing
> > > > > available
> > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > >> >`table.exec.emit.early-fire.de <
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > >> >**lay`.
> > > > > >> >Although their documentation states that they are only
> applicable
> > > to
> > > > > the
> > > > > >> >Window operator, it seems possible that they may also be
> relevant
> > > in
> > > > > the
> > > > > >> >context of this FLIP. Otherwise, having different early firing
> > > > > behaviors
> > > > > >> >for different operators could confuse users.*
> > > > > >> >
> > > > > >> > - +1 on unifying early-fire behaviors to avoid confusion.
> I’ve
> > > > > added a
> > > > > >> > section
> > > > > >> > <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > >> >
> > > > > >> >in
> > > > > >> > the proposal highlighting that we should align hint-based
> > > interval
> > > > > join
> > > > > >> > configurations with the existing table.exec.emit.* settings.
> > > > > >> Suggestions
> > > > > >> > on how to make the unification are welcome! We plan to
> extend
> > > > early
> > > > > >> firing
> > > > > >> > to window joins via hints in a future FLIP.
> > > > > >> >
> > > > > >> >
> > > > > >> >*> 2. The design of `time_mode` is excellent. Similar to point
> 1,
> > > > > perhaps
> > > > > >> >we can introduce it to other window-related operators in the
> > > future.>
> > > > > 3.
> > > > > >> >You need to modify the FlinkChangelogModeInferenceProgram to
> > ensure
> > > > > that
> > > > > >> >downstream nodes of interval joins with early firing enabled
> are
> > > > aware
> > > > > of
> > > > > >> >retract or upsert messages.*
> > > > > >> >
> > > > > >> > - We agree that time_mode could be introduced to other
> > > > window-based
> > > > > >> > operators down the road. We also want to support early fire
> > for
> > > > > >> > window join. Also, thanks for highlighting
> > > > > >> > FlinkChangelogModeInferenceProgram! I added the code change
> on
> > > it
> > > > > >> > <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > >> >
> > > > > >> > in the proposal.
> > > > > >> >
> > > > > >> >
> > > > > >> >Thanks again for your time and feedback! I’ve updated the
> > proposal
> > > > with
> > > > > >> >these points. Please let me know if there’s anything else I
> > should
> > > > > >> address.
> > > > > >> >
> > > > > >> >Best,
> > > > > >> >Weiqing
> > > > > >> >
> > > > > >> >
> > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com>
> > wrote:
> > > > > >> >
> > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> > > > > questions:
> > > > > >> >>
> > > > > >> >> 1. Currently, there are some configs related to early firing
> > > > > available
> > > > > >> to
> > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > >> >>
> > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > documentation
> > > > > states
> > > > > >> >> that they are only applicable to the Window operator,
> > > > > >> >>
> > > > > >> >> it seems possible that they may also be relevant in the
> context
> > > of
> > > > > this
> > > > > >> >> FLIP. Otherwise, having different early firing behaviors
> > > > > >> >>
> > > > > >> >> for different operators could confuse users.
> > > > > >> >>
> > > > > >> >> 2. The design of `time_mode` is excellent. Similar to point
> 1,
> > > > > perhaps
> > > > > >> we
> > > > > >> >> can introduce it to other window-related operators
> > > > > >> >>
> > > > > >> >> in the future.
> > > > > >> >>
> > > > > >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram
> to
> > > > > ensure
> > > > > >> >> that downstream nodes of interval joins with
> > > > > >> >>
> > > > > >> >> early firing enabled are aware of retract or upsert messages.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >>
> > > > > >> >> Best！
> > > > > >> >> Xuyang
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com>
> > > wrote:
> > > > > >> >> >Hi Weiqing,
> > > > > >> >> >
> > > > > >> >> >Thanks for the proposal. IMO, adding early fire for time
> > > interval
> > > > > outer
> > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > >> >> >
> > > > > >> >> >1. The current output stream of a time interval outer join
> is
> > an
> > > > > >> >> >append-only stream. This change will make it a potential
> > > > retractable
> > > > > >> >> >stream. I'm not sure if the planner supports a dynamic
> output
> > > type
> > > > > like
> > > > > >> >> >that. Could you add this part to your design doc?
> > > > > >> >> >2. What's the use case when the downstream components need
> to
> > > get
> > > > > the
> > > > > >> >> early
> > > > > >> >> >fired results regularly?
> > > > > >> >> >3. The time interval join operator itself is not quite
> > efficient
> > > > > when
> > > > > >> the
> > > > > >> >> >state becomes large. Will there be any extra overhead after
> > > > > introducing
> > > > > >> >> >this feature?
> > > > > >> >> >
> > > > > >> >> >Thanks,
> > > > > >> >> >Xingcan
> > > > > >> >> >
> > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > yangweiqing001@gmail.com>
> > > > > >> >> >wrote:
> > > > > >> >> >
> > > > > >> >> >> Hi all,
> > > > > >> >> >>
> > > > > >> >> >> Just a gentle reminder regarding the proposal I shared on
> > > early
> > > > > fire
> > > > > >> >> >> support for Flink SQL interval joins. I’d greatly
> appreciate
> > > > your
> > > > > >> >> feedback
> > > > > >> >> >> or suggestions.
> > > > > >> >> >>
> > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > >> >> >> <
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >> Thank you!
> > > > > >> >> >>
> > > > > >> >> >> Best,
> > > > > >> >> >> Weiqing
> > > > > >> >> >>
> > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > >> yangweiqing001@gmail.com
> > > > > >> >> >
> > > > > >> >> >> wrote:
> > > > > >> >> >>
> > > > > >> >> >> > Hi all,
> > > > > >> >> >> >
> > > > > >> >> >> > I’d like to initiate a discussion about introducing
> early
> > > fire
> > > > > >> support
> > > > > >> >> >> for
> > > > > >> >> >> > Flink SQL interval joins.
> > > > > >> >> >> >
> > > > > >> >> >> > *Motivation*
> > > > > >> >> >> > In many streaming applications, particularly real-time
> > > > analytics
> > > > > >> and
> > > > > >> >> >> > monitoring systems, it is valuable to obtain partial
> > results
> > > > > >> earlier
> > > > > >> >> >> rather
> > > > > >> >> >> > than waiting for full join conditions to finalize. For
> > Flink
> > > > SQL
> > > > > >> >> interval
> > > > > >> >> >> > joins, results are typically delayed until watermarks
> > ensure
> > > > no
> > > > > >> more
> > > > > >> >> >> > matches can occur. This delay can be challenging for
> > > scenarios
> > > > > that
> > > > > >> >> >> require
> > > > > >> >> >> > fast feedback. Early fire support addresses this by
> > emitting
> > > > > >> >> intermediate
> > > > > >> >> >> > results speculatively and using retractions or updates
> to
> > > > > maintain
> > > > > >> >> >> eventual
> > > > > >> >> >> > consistency and ensure correctness.
> > > > > >> >> >> >
> > > > > >> >> >> > Here’s the proposal document: Link
> > > > > >> >> >> > <
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >> > Your feedback and ideas are welcome to refine this
> > feature.
> > > > > >> >> >> >
> > > > > >> >> >> > Thanks,
> > > > > >> >> >> > Weiqing
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Thanks for the response!
> If the optional `interval` in the proposal is enabled, early-fire outputs
will occur repeatedly at every earlyFireInterval, not just once. After that
interval concludes, there will also be a final emission.
One more question, would it make sense to support a mechanism through
configuration where only one earlyFire is fired for an interval or window?
This would also simplify the state overhead for use cases where only one
early fire is required within a window or interval. Thoughts?
Regards
Venkata krishnan
On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi Venkata,
>
> Thanks for your feedback!
>
> If the optional `interval` in the proposal is enabled, early-fire outputs
> will occur repeatedly at every earlyFireInterval, not just once. After that
> interval concludes, there will also be a final emission.
>
> We haven’t included a late-fire mechanism for interval joins in this FLIP,
> but it’s certainly something we can consider for future efforts!
>
> Best regards,
> Weiqing
>
> On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> vsowrira@asu.edu>
> wrote:
>
> > Weiqing,
> >
> > Thanks, this is a great addition to Flink SQL. Also instead of
> controlling
> > and configuring through Flink configs unlike the older window
> aggregation,
> > hints seems to be a much better approach. This enables a customizable
> early
> > fire behavior for individual interval joins.
> >
> > Couple of questions:
> >
> > 1. Does the *early fire* emit an output every earlyFireInterval time or
> > will it be a one time output emission and another output emitted at the
> end
> > of the interval?
> > 2. Are there plans to support *late fire *similar to the window
> > aggregations in later FLIPs?
> >
> > Regards
> > Venkata krishnan
> >
> >
> > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <ya...@gmail.com>
> > wrote:
> >
> > > Thanks for reviewing, Xuyang!
> > >
> > > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> > >
> > > If no further objections arise from anyone, I’ll proceed to mark FLIP
> as
> > > ready for voting.
> > >
> > > Best regards,
> > > Weiqing
> > >
> > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
> > >
> > > > LGTM overall. Thanks for updating. I have no problem and +1 for this
> > > > feature.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best！
> > > > Xuyang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com> 写道：
> > > > >Hi Xuyang,
> > > > >
> > > > >Thank you for your detailed feedback! I’ve updated the proposal doc
> > > > >accordingly. Please feel free to take another look and let me know
> if
> > > you
> > > > >have any further thoughts or suggestions.
> > > > >
> > > > >Best regards,
> > > > >Weiqing
> > > > >
> > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
> > > > >
> > > > >> Hi, Weiqing.
> > > > >>
> > > > >> After reading the new FLIP, I have no issues with the part `public
> > > > >> interface`. I only have some questions regarding
> > > > >>
> > > > >> the details in the Proposed Changes section.
> > > > >>
> > > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin node,
> > > IIUC:
> > > > >>
> > > > >> - When early firing is enabled, the UpdateKind of the IntervalJoin
> > can
> > > > be
> > > > >> either ONLY_UPDATE_AFTER or
> > > > >>
> > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> requirements
> > of
> > > > the
> > > > >> sink. And the ModifyKind is always ALL.
> > > > >>
> > > > >> - When early firing is disabled, the UpdateKind of the
> IntervalJoin
> > is
> > > > >> NONE, and the ModifyKind is INSERT.
> > > > >>
> > > > >> - Nevertheless, whether early firing is enabled or disabled, the
> > > > >> IntervalJoin should always require its input to keep
> > > > >>
> > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Best！
> > > > >> Xuyang
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com>
> > > > wrote:
> > > > >> >Hi Xingcan and Xuyang,
> > > > >> >
> > > > >> >Thanks so much for the feedback - it was very helpful!
> > > > >> >
> > > > >> >*> 1. The current output stream of a time interval outer join is
> an
> > > > >> >append-only stream. This change will make it a potential
> > retractable
> > > > >> >stream. I'm not sure if the planner supports a dynamic output
> type
> > > like
> > > > >> >that. Could you add this part to your design doc?*
> > > > >> >
> > > > >> >
> > > > >> > - Yes, enabling early firing on time interval outer joins can
> > emit
> > > > >> > retractions when previously emitted rows are updated or
> > > invalidated
> > > > by
> > > > >> > later matches. I’ve updated the proposal (Planner Awareness
> > > > >> > <
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > >> >
> > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > >> > <
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > >> >)
> > > > >> > to clarify that the stream might switch from append-only to a
> > > > >> > retract/upsert stream. Let me know if anything is missing.
> > > > >> >
> > > > >> >
> > > > >> >*> 2. What's the use case when the downstream components need to
> > get
> > > > the
> > > > >> >early fired results regularly?*
> > > > >> >
> > > > >> >
> > > > >> > - The new INTERVAL option (in addition to DELAY) allows
> periodic
> > > > >> updates
> > > > >> > (e.g., every 10 minutes) after the initial delay. This
> captures
> > > how
> > > > >> results
> > > > >> > evolve over time, similar to Apache Beam’s “Repeatedly”
> option.
> > > > >> >
> > > > >> >
> > > > >> >*> 3. The time interval join operator itself is not quite
> efficient
> > > > when
> > > > >> >the state becomes large. Will there be any extra overhead after
> > > > >> introducing
> > > > >> >this feature?*
> > > > >> >
> > > > >> > - Early fire does introduce some overhead by potentially
> > emitting
> > > > >> > partial matches multiple times with retraction (avoiding
> > duplicate
> > > > >> outputs
> > > > >> > though). However, if it’s disabled, there is no additional
> cost.
> > > > Most
> > > > >> users
> > > > >> > find the performance trade-off acceptable for the real-time
> > > > insights it
> > > > >> > provides.
> > > > >> >
> > > > >> >
> > > > >> >*> 1. Currently, there are some configs related to early firing
> > > > available
> > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > >> >`table.exec.emit.early-fire.de <
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > >> >**lay`.
> > > > >> >Although their documentation states that they are only applicable
> > to
> > > > the
> > > > >> >Window operator, it seems possible that they may also be relevant
> > in
> > > > the
> > > > >> >context of this FLIP. Otherwise, having different early firing
> > > > behaviors
> > > > >> >for different operators could confuse users.*
> > > > >> >
> > > > >> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve
> > > > added a
> > > > >> > section
> > > > >> > <
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > >> >
> > > > >> >in
> > > > >> > the proposal highlighting that we should align hint-based
> > interval
> > > > join
> > > > >> > configurations with the existing table.exec.emit.* settings.
> > > > >> Suggestions
> > > > >> > on how to make the unification are welcome! We plan to extend
> > > early
> > > > >> firing
> > > > >> > to window joins via hints in a future FLIP.
> > > > >> >
> > > > >> >
> > > > >> >*> 2. The design of `time_mode` is excellent. Similar to point 1,
> > > > perhaps
> > > > >> >we can introduce it to other window-related operators in the
> > future.>
> > > > 3.
> > > > >> >You need to modify the FlinkChangelogModeInferenceProgram to
> ensure
> > > > that
> > > > >> >downstream nodes of interval joins with early firing enabled are
> > > aware
> > > > of
> > > > >> >retract or upsert messages.*
> > > > >> >
> > > > >> > - We agree that time_mode could be introduced to other
> > > window-based
> > > > >> > operators down the road. We also want to support early fire
> for
> > > > >> > window join. Also, thanks for highlighting
> > > > >> > FlinkChangelogModeInferenceProgram! I added the code change on
> > it
> > > > >> > <
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > >> >
> > > > >> > in the proposal.
> > > > >> >
> > > > >> >
> > > > >> >Thanks again for your time and feedback! I’ve updated the
> proposal
> > > with
> > > > >> >these points. Please let me know if there’s anything else I
> should
> > > > >> address.
> > > > >> >
> > > > >> >Best,
> > > > >> >Weiqing
> > > > >> >
> > > > >> >
> > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com>
> wrote:
> > > > >> >
> > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> > > > questions:
> > > > >> >>
> > > > >> >> 1. Currently, there are some configs related to early firing
> > > > available
> > > > >> to
> > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > >> >>
> > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> documentation
> > > > states
> > > > >> >> that they are only applicable to the Window operator,
> > > > >> >>
> > > > >> >> it seems possible that they may also be relevant in the context
> > of
> > > > this
> > > > >> >> FLIP. Otherwise, having different early firing behaviors
> > > > >> >>
> > > > >> >> for different operators could confuse users.
> > > > >> >>
> > > > >> >> 2. The design of `time_mode` is excellent. Similar to point 1,
> > > > perhaps
> > > > >> we
> > > > >> >> can introduce it to other window-related operators
> > > > >> >>
> > > > >> >> in the future.
> > > > >> >>
> > > > >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to
> > > > ensure
> > > > >> >> that downstream nodes of interval joins with
> > > > >> >>
> > > > >> >> early firing enabled are aware of retract or upsert messages.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >>
> > > > >> >> Best！
> > > > >> >> Xuyang
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com>
> > wrote:
> > > > >> >> >Hi Weiqing,
> > > > >> >> >
> > > > >> >> >Thanks for the proposal. IMO, adding early fire for time
> > interval
> > > > outer
> > > > >> >> >joins is feasible overall. I have a few questions.
> > > > >> >> >
> > > > >> >> >1. The current output stream of a time interval outer join is
> an
> > > > >> >> >append-only stream. This change will make it a potential
> > > retractable
> > > > >> >> >stream. I'm not sure if the planner supports a dynamic output
> > type
> > > > like
> > > > >> >> >that. Could you add this part to your design doc?
> > > > >> >> >2. What's the use case when the downstream components need to
> > get
> > > > the
> > > > >> >> early
> > > > >> >> >fired results regularly?
> > > > >> >> >3. The time interval join operator itself is not quite
> efficient
> > > > when
> > > > >> the
> > > > >> >> >state becomes large. Will there be any extra overhead after
> > > > introducing
> > > > >> >> >this feature?
> > > > >> >> >
> > > > >> >> >Thanks,
> > > > >> >> >Xingcan
> > > > >> >> >
> > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com>
> > > > >> >> >wrote:
> > > > >> >> >
> > > > >> >> >> Hi all,
> > > > >> >> >>
> > > > >> >> >> Just a gentle reminder regarding the proposal I shared on
> > early
> > > > fire
> > > > >> >> >> support for Flink SQL interval joins. I’d greatly appreciate
> > > your
> > > > >> >> feedback
> > > > >> >> >> or suggestions.
> > > > >> >> >>
> > > > >> >> >> Here’s the link to the proposal document: Link
> > > > >> >> >> <
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >> Thank you!
> > > > >> >> >>
> > > > >> >> >> Best,
> > > > >> >> >> Weiqing
> > > > >> >> >>
> > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > >> yangweiqing001@gmail.com
> > > > >> >> >
> > > > >> >> >> wrote:
> > > > >> >> >>
> > > > >> >> >> > Hi all,
> > > > >> >> >> >
> > > > >> >> >> > I’d like to initiate a discussion about introducing early
> > fire
> > > > >> support
> > > > >> >> >> for
> > > > >> >> >> > Flink SQL interval joins.
> > > > >> >> >> >
> > > > >> >> >> > *Motivation*
> > > > >> >> >> > In many streaming applications, particularly real-time
> > > analytics
> > > > >> and
> > > > >> >> >> > monitoring systems, it is valuable to obtain partial
> results
> > > > >> earlier
> > > > >> >> >> rather
> > > > >> >> >> > than waiting for full join conditions to finalize. For
> Flink
> > > SQL
> > > > >> >> interval
> > > > >> >> >> > joins, results are typically delayed until watermarks
> ensure
> > > no
> > > > >> more
> > > > >> >> >> > matches can occur. This delay can be challenging for
> > scenarios
> > > > that
> > > > >> >> >> require
> > > > >> >> >> > fast feedback. Early fire support addresses this by
> emitting
> > > > >> >> intermediate
> > > > >> >> >> > results speculatively and using retractions or updates to
> > > > maintain
> > > > >> >> >> eventual
> > > > >> >> >> > consistency and ensure correctness.
> > > > >> >> >> >
> > > > >> >> >> > Here’s the proposal document: Link
> > > > >> >> >> > <
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > Your feedback and ideas are welcome to refine this
> feature.
> > > > >> >> >> >
> > > > >> >> >> > Thanks,
> > > > >> >> >> > Weiqing
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Venkata,
Thanks for your feedback!
If the optional `interval` in the proposal is enabled, early-fire outputs
will occur repeatedly at every earlyFireInterval, not just once. After that
interval concludes, there will also be a final emission.
We haven’t included a late-fire mechanism for interval joins in this FLIP,
but it’s certainly something we can consider for future efforts!
Best regards,
Weiqing
On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <vs...@asu.edu>
wrote:
> Weiqing,
>
> Thanks, this is a great addition to Flink SQL. Also instead of controlling
> and configuring through Flink configs unlike the older window aggregation,
> hints seems to be a much better approach. This enables a customizable early
> fire behavior for individual interval joins.
>
> Couple of questions:
>
> 1. Does the *early fire* emit an output every earlyFireInterval time or
> will it be a one time output emission and another output emitted at the end
> of the interval?
> 2. Are there plans to support *late fire *similar to the window
> aggregations in later FLIPs?
>
> Regards
> Venkata krishnan
>
>
> On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Thanks for reviewing, Xuyang!
> >
> > Xingcan (@xingcanc@gmail.com) – do you have any concerns?
> >
> > If no further objections arise from anyone, I’ll proceed to mark FLIP as
> > ready for voting.
> >
> > Best regards,
> > Weiqing
> >
> > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
> >
> > > LGTM overall. Thanks for updating. I have no problem and +1 for this
> > > feature.
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Best！
> > > Xuyang
> > >
> > >
> > >
> > >
> > >
> > > 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com> 写道：
> > > >Hi Xuyang,
> > > >
> > > >Thank you for your detailed feedback! I’ve updated the proposal doc
> > > >accordingly. Please feel free to take another look and let me know if
> > you
> > > >have any further thoughts or suggestions.
> > > >
> > > >Best regards,
> > > >Weiqing
> > > >
> > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
> > > >
> > > >> Hi, Weiqing.
> > > >>
> > > >> After reading the new FLIP, I have no issues with the part `public
> > > >> interface`. I only have some questions regarding
> > > >>
> > > >> the details in the Proposed Changes section.
> > > >>
> > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin node,
> > IIUC:
> > > >>
> > > >> - When early firing is enabled, the UpdateKind of the IntervalJoin
> can
> > > be
> > > >> either ONLY_UPDATE_AFTER or
> > > >>
> > > >> degrade to BEFORE_AND_AFTER, depending entirely on the requirements
> of
> > > the
> > > >> sink. And the ModifyKind is always ALL.
> > > >>
> > > >> - When early firing is disabled, the UpdateKind of the IntervalJoin
> is
> > > >> NONE, and the ModifyKind is INSERT.
> > > >>
> > > >> - Nevertheless, whether early firing is enabled or disabled, the
> > > >> IntervalJoin should always require its input to keep
> > > >>
> > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Best！
> > > >> Xuyang
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com>
> > > wrote:
> > > >> >Hi Xingcan and Xuyang,
> > > >> >
> > > >> >Thanks so much for the feedback - it was very helpful!
> > > >> >
> > > >> >*> 1. The current output stream of a time interval outer join is an
> > > >> >append-only stream. This change will make it a potential
> retractable
> > > >> >stream. I'm not sure if the planner supports a dynamic output type
> > like
> > > >> >that. Could you add this part to your design doc?*
> > > >> >
> > > >> >
> > > >> > - Yes, enabling early firing on time interval outer joins can
> emit
> > > >> > retractions when previously emitted rows are updated or
> > invalidated
> > > by
> > > >> > later matches. I’ve updated the proposal (Planner Awareness
> > > >> > <
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > >> >
> > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > >> > <
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > >> >)
> > > >> > to clarify that the stream might switch from append-only to a
> > > >> > retract/upsert stream. Let me know if anything is missing.
> > > >> >
> > > >> >
> > > >> >*> 2. What's the use case when the downstream components need to
> get
> > > the
> > > >> >early fired results regularly?*
> > > >> >
> > > >> >
> > > >> > - The new INTERVAL option (in addition to DELAY) allows periodic
> > > >> updates
> > > >> > (e.g., every 10 minutes) after the initial delay. This captures
> > how
> > > >> results
> > > >> > evolve over time, similar to Apache Beam’s “Repeatedly” option.
> > > >> >
> > > >> >
> > > >> >*> 3. The time interval join operator itself is not quite efficient
> > > when
> > > >> >the state becomes large. Will there be any extra overhead after
> > > >> introducing
> > > >> >this feature?*
> > > >> >
> > > >> > - Early fire does introduce some overhead by potentially
> emitting
> > > >> > partial matches multiple times with retraction (avoiding
> duplicate
> > > >> outputs
> > > >> > though). However, if it’s disabled, there is no additional cost.
> > > Most
> > > >> users
> > > >> > find the performance trade-off acceptable for the real-time
> > > insights it
> > > >> > provides.
> > > >> >
> > > >> >
> > > >> >*> 1. Currently, there are some configs related to early firing
> > > available
> > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > >> >`table.exec.emit.early-fire.de <
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > >> >**lay`.
> > > >> >Although their documentation states that they are only applicable
> to
> > > the
> > > >> >Window operator, it seems possible that they may also be relevant
> in
> > > the
> > > >> >context of this FLIP. Otherwise, having different early firing
> > > behaviors
> > > >> >for different operators could confuse users.*
> > > >> >
> > > >> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve
> > > added a
> > > >> > section
> > > >> > <
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > >> >
> > > >> >in
> > > >> > the proposal highlighting that we should align hint-based
> interval
> > > join
> > > >> > configurations with the existing table.exec.emit.* settings.
> > > >> Suggestions
> > > >> > on how to make the unification are welcome! We plan to extend
> > early
> > > >> firing
> > > >> > to window joins via hints in a future FLIP.
> > > >> >
> > > >> >
> > > >> >*> 2. The design of `time_mode` is excellent. Similar to point 1,
> > > perhaps
> > > >> >we can introduce it to other window-related operators in the
> future.>
> > > 3.
> > > >> >You need to modify the FlinkChangelogModeInferenceProgram to ensure
> > > that
> > > >> >downstream nodes of interval joins with early firing enabled are
> > aware
> > > of
> > > >> >retract or upsert messages.*
> > > >> >
> > > >> > - We agree that time_mode could be introduced to other
> > window-based
> > > >> > operators down the road. We also want to support early fire for
> > > >> > window join. Also, thanks for highlighting
> > > >> > FlinkChangelogModeInferenceProgram! I added the code change on
> it
> > > >> > <
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > >> >
> > > >> > in the proposal.
> > > >> >
> > > >> >
> > > >> >Thanks again for your time and feedback! I’ve updated the proposal
> > with
> > > >> >these points. Please let me know if there’s anything else I should
> > > >> address.
> > > >> >
> > > >> >Best,
> > > >> >Weiqing
> > > >> >
> > > >> >
> > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
> > > >> >
> > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> > > questions:
> > > >> >>
> > > >> >> 1. Currently, there are some configs related to early firing
> > > available
> > > >> to
> > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > >> >>
> > > >> >> `table.exec.emit.early-fire.delay`. Although their documentation
> > > states
> > > >> >> that they are only applicable to the Window operator,
> > > >> >>
> > > >> >> it seems possible that they may also be relevant in the context
> of
> > > this
> > > >> >> FLIP. Otherwise, having different early firing behaviors
> > > >> >>
> > > >> >> for different operators could confuse users.
> > > >> >>
> > > >> >> 2. The design of `time_mode` is excellent. Similar to point 1,
> > > perhaps
> > > >> we
> > > >> >> can introduce it to other window-related operators
> > > >> >>
> > > >> >> in the future.
> > > >> >>
> > > >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to
> > > ensure
> > > >> >> that downstream nodes of interval joins with
> > > >> >>
> > > >> >> early firing enabled are aware of retract or upsert messages.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >>
> > > >> >> Best！
> > > >> >> Xuyang
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com>
> wrote:
> > > >> >> >Hi Weiqing,
> > > >> >> >
> > > >> >> >Thanks for the proposal. IMO, adding early fire for time
> interval
> > > outer
> > > >> >> >joins is feasible overall. I have a few questions.
> > > >> >> >
> > > >> >> >1. The current output stream of a time interval outer join is an
> > > >> >> >append-only stream. This change will make it a potential
> > retractable
> > > >> >> >stream. I'm not sure if the planner supports a dynamic output
> type
> > > like
> > > >> >> >that. Could you add this part to your design doc?
> > > >> >> >2. What's the use case when the downstream components need to
> get
> > > the
> > > >> >> early
> > > >> >> >fired results regularly?
> > > >> >> >3. The time interval join operator itself is not quite efficient
> > > when
> > > >> the
> > > >> >> >state becomes large. Will there be any extra overhead after
> > > introducing
> > > >> >> >this feature?
> > > >> >> >
> > > >> >> >Thanks,
> > > >> >> >Xingcan
> > > >> >> >
> > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > yangweiqing001@gmail.com>
> > > >> >> >wrote:
> > > >> >> >
> > > >> >> >> Hi all,
> > > >> >> >>
> > > >> >> >> Just a gentle reminder regarding the proposal I shared on
> early
> > > fire
> > > >> >> >> support for Flink SQL interval joins. I’d greatly appreciate
> > your
> > > >> >> feedback
> > > >> >> >> or suggestions.
> > > >> >> >>
> > > >> >> >> Here’s the link to the proposal document: Link
> > > >> >> >> <
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >> Thank you!
> > > >> >> >>
> > > >> >> >> Best,
> > > >> >> >> Weiqing
> > > >> >> >>
> > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > >> yangweiqing001@gmail.com
> > > >> >> >
> > > >> >> >> wrote:
> > > >> >> >>
> > > >> >> >> > Hi all,
> > > >> >> >> >
> > > >> >> >> > I’d like to initiate a discussion about introducing early
> fire
> > > >> support
> > > >> >> >> for
> > > >> >> >> > Flink SQL interval joins.
> > > >> >> >> >
> > > >> >> >> > *Motivation*
> > > >> >> >> > In many streaming applications, particularly real-time
> > analytics
> > > >> and
> > > >> >> >> > monitoring systems, it is valuable to obtain partial results
> > > >> earlier
> > > >> >> >> rather
> > > >> >> >> > than waiting for full join conditions to finalize. For Flink
> > SQL
> > > >> >> interval
> > > >> >> >> > joins, results are typically delayed until watermarks ensure
> > no
> > > >> more
> > > >> >> >> > matches can occur. This delay can be challenging for
> scenarios
> > > that
> > > >> >> >> require
> > > >> >> >> > fast feedback. Early fire support addresses this by emitting
> > > >> >> intermediate
> > > >> >> >> > results speculatively and using retractions or updates to
> > > maintain
> > > >> >> >> eventual
> > > >> >> >> > consistency and ensure correctness.
> > > >> >> >> >
> > > >> >> >> > Here’s the proposal document: Link
> > > >> >> >> > <
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > Your feedback and ideas are welcome to refine this feature.
> > > >> >> >> >
> > > >> >> >> > Thanks,
> > > >> >> >> > Weiqing
> > > >> >> >> >
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Weiqing,
Thanks, this is a great addition to Flink SQL. Also instead of controlling
and configuring through Flink configs unlike the older window aggregation,
hints seems to be a much better approach. This enables a customizable early
fire behavior for individual interval joins.
Couple of questions:
1. Does the *early fire* emit an output every earlyFireInterval time or
will it be a one time output emission and another output emitted at the end
of the interval?
2. Are there plans to support *late fire *similar to the window
aggregations in later FLIPs?
Regards
Venkata krishnan
On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Thanks for reviewing, Xuyang!
>
> Xingcan (@xingcanc@gmail.com) – do you have any concerns?
>
> If no further objections arise from anyone, I’ll proceed to mark FLIP as
> ready for voting.
>
> Best regards,
> Weiqing
>
> On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
>
> > LGTM overall. Thanks for updating. I have no problem and +1 for this
> > feature.
> >
> >
> >
> >
> > --
> >
> > Best！
> > Xuyang
> >
> >
> >
> >
> >
> > 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com> 写道：
> > >Hi Xuyang,
> > >
> > >Thank you for your detailed feedback! I’ve updated the proposal doc
> > >accordingly. Please feel free to take another look and let me know if
> you
> > >have any further thoughts or suggestions.
> > >
> > >Best regards,
> > >Weiqing
> > >
> > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
> > >
> > >> Hi, Weiqing.
> > >>
> > >> After reading the new FLIP, I have no issues with the part `public
> > >> interface`. I only have some questions regarding
> > >>
> > >> the details in the Proposed Changes section.
> > >>
> > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin node,
> IIUC:
> > >>
> > >> - When early firing is enabled, the UpdateKind of the IntervalJoin can
> > be
> > >> either ONLY_UPDATE_AFTER or
> > >>
> > >> degrade to BEFORE_AND_AFTER, depending entirely on the requirements of
> > the
> > >> sink. And the ModifyKind is always ALL.
> > >>
> > >> - When early firing is disabled, the UpdateKind of the IntervalJoin is
> > >> NONE, and the ModifyKind is INSERT.
> > >>
> > >> - Nevertheless, whether early firing is enabled or disabled, the
> > >> IntervalJoin should always require its input to keep
> > >>
> > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Best！
> > >> Xuyang
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com>
> > wrote:
> > >> >Hi Xingcan and Xuyang,
> > >> >
> > >> >Thanks so much for the feedback - it was very helpful!
> > >> >
> > >> >*> 1. The current output stream of a time interval outer join is an
> > >> >append-only stream. This change will make it a potential retractable
> > >> >stream. I'm not sure if the planner supports a dynamic output type
> like
> > >> >that. Could you add this part to your design doc?*
> > >> >
> > >> >
> > >> > - Yes, enabling early firing on time interval outer joins can emit
> > >> > retractions when previously emitted rows are updated or
> invalidated
> > by
> > >> > later matches. I’ve updated the proposal (Planner Awareness
> > >> > <
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > >> >
> > >> > and Changes in FlinkChangelogModeInferenceProgram
> > >> > <
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > >> >)
> > >> > to clarify that the stream might switch from append-only to a
> > >> > retract/upsert stream. Let me know if anything is missing.
> > >> >
> > >> >
> > >> >*> 2. What's the use case when the downstream components need to get
> > the
> > >> >early fired results regularly?*
> > >> >
> > >> >
> > >> > - The new INTERVAL option (in addition to DELAY) allows periodic
> > >> updates
> > >> > (e.g., every 10 minutes) after the initial delay. This captures
> how
> > >> results
> > >> > evolve over time, similar to Apache Beam’s “Repeatedly” option.
> > >> >
> > >> >
> > >> >*> 3. The time interval join operator itself is not quite efficient
> > when
> > >> >the state becomes large. Will there be any extra overhead after
> > >> introducing
> > >> >this feature?*
> > >> >
> > >> > - Early fire does introduce some overhead by potentially emitting
> > >> > partial matches multiple times with retraction (avoiding duplicate
> > >> outputs
> > >> > though). However, if it’s disabled, there is no additional cost.
> > Most
> > >> users
> > >> > find the performance trade-off acceptable for the real-time
> > insights it
> > >> > provides.
> > >> >
> > >> >
> > >> >*> 1. Currently, there are some configs related to early firing
> > available
> > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > >> >`table.exec.emit.early-fire.de <
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > >> >**lay`.
> > >> >Although their documentation states that they are only applicable to
> > the
> > >> >Window operator, it seems possible that they may also be relevant in
> > the
> > >> >context of this FLIP. Otherwise, having different early firing
> > behaviors
> > >> >for different operators could confuse users.*
> > >> >
> > >> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve
> > added a
> > >> > section
> > >> > <
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > >> >
> > >> >in
> > >> > the proposal highlighting that we should align hint-based interval
> > join
> > >> > configurations with the existing table.exec.emit.* settings.
> > >> Suggestions
> > >> > on how to make the unification are welcome! We plan to extend
> early
> > >> firing
> > >> > to window joins via hints in a future FLIP.
> > >> >
> > >> >
> > >> >*> 2. The design of `time_mode` is excellent. Similar to point 1,
> > perhaps
> > >> >we can introduce it to other window-related operators in the future.>
> > 3.
> > >> >You need to modify the FlinkChangelogModeInferenceProgram to ensure
> > that
> > >> >downstream nodes of interval joins with early firing enabled are
> aware
> > of
> > >> >retract or upsert messages.*
> > >> >
> > >> > - We agree that time_mode could be introduced to other
> window-based
> > >> > operators down the road. We also want to support early fire for
> > >> > window join. Also, thanks for highlighting
> > >> > FlinkChangelogModeInferenceProgram! I added the code change on it
> > >> > <
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > >> >
> > >> > in the proposal.
> > >> >
> > >> >
> > >> >Thanks again for your time and feedback! I’ve updated the proposal
> with
> > >> >these points. Please let me know if there’s anything else I should
> > >> address.
> > >> >
> > >> >Best,
> > >> >Weiqing
> > >> >
> > >> >
> > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
> > >> >
> > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> > questions:
> > >> >>
> > >> >> 1. Currently, there are some configs related to early firing
> > available
> > >> to
> > >> >> users: `table.exec.emit.early-fire.enabled` and
> > >> >>
> > >> >> `table.exec.emit.early-fire.delay`. Although their documentation
> > states
> > >> >> that they are only applicable to the Window operator,
> > >> >>
> > >> >> it seems possible that they may also be relevant in the context of
> > this
> > >> >> FLIP. Otherwise, having different early firing behaviors
> > >> >>
> > >> >> for different operators could confuse users.
> > >> >>
> > >> >> 2. The design of `time_mode` is excellent. Similar to point 1,
> > perhaps
> > >> we
> > >> >> can introduce it to other window-related operators
> > >> >>
> > >> >> in the future.
> > >> >>
> > >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to
> > ensure
> > >> >> that downstream nodes of interval joins with
> > >> >>
> > >> >> early firing enabled are aware of retract or upsert messages.
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >>
> > >> >> Best！
> > >> >> Xuyang
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
> > >> >> >Hi Weiqing,
> > >> >> >
> > >> >> >Thanks for the proposal. IMO, adding early fire for time interval
> > outer
> > >> >> >joins is feasible overall. I have a few questions.
> > >> >> >
> > >> >> >1. The current output stream of a time interval outer join is an
> > >> >> >append-only stream. This change will make it a potential
> retractable
> > >> >> >stream. I'm not sure if the planner supports a dynamic output type
> > like
> > >> >> >that. Could you add this part to your design doc?
> > >> >> >2. What's the use case when the downstream components need to get
> > the
> > >> >> early
> > >> >> >fired results regularly?
> > >> >> >3. The time interval join operator itself is not quite efficient
> > when
> > >> the
> > >> >> >state becomes large. Will there be any extra overhead after
> > introducing
> > >> >> >this feature?
> > >> >> >
> > >> >> >Thanks,
> > >> >> >Xingcan
> > >> >> >
> > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > yangweiqing001@gmail.com>
> > >> >> >wrote:
> > >> >> >
> > >> >> >> Hi all,
> > >> >> >>
> > >> >> >> Just a gentle reminder regarding the proposal I shared on early
> > fire
> > >> >> >> support for Flink SQL interval joins. I’d greatly appreciate
> your
> > >> >> feedback
> > >> >> >> or suggestions.
> > >> >> >>
> > >> >> >> Here’s the link to the proposal document: Link
> > >> >> >> <
> > >> >> >>
> > >> >>
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > >> >> >> >
> > >> >> >>
> > >> >> >> Thank you!
> > >> >> >>
> > >> >> >> Best,
> > >> >> >> Weiqing
> > >> >> >>
> > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > >> yangweiqing001@gmail.com
> > >> >> >
> > >> >> >> wrote:
> > >> >> >>
> > >> >> >> > Hi all,
> > >> >> >> >
> > >> >> >> > I’d like to initiate a discussion about introducing early fire
> > >> support
> > >> >> >> for
> > >> >> >> > Flink SQL interval joins.
> > >> >> >> >
> > >> >> >> > *Motivation*
> > >> >> >> > In many streaming applications, particularly real-time
> analytics
> > >> and
> > >> >> >> > monitoring systems, it is valuable to obtain partial results
> > >> earlier
> > >> >> >> rather
> > >> >> >> > than waiting for full join conditions to finalize. For Flink
> SQL
> > >> >> interval
> > >> >> >> > joins, results are typically delayed until watermarks ensure
> no
> > >> more
> > >> >> >> > matches can occur. This delay can be challenging for scenarios
> > that
> > >> >> >> require
> > >> >> >> > fast feedback. Early fire support addresses this by emitting
> > >> >> intermediate
> > >> >> >> > results speculatively and using retractions or updates to
> > maintain
> > >> >> >> eventual
> > >> >> >> > consistency and ensure correctness.
> > >> >> >> >
> > >> >> >> > Here’s the proposal document: Link
> > >> >> >> > <
> > >> >> >>
> > >> >>
> > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > Your feedback and ideas are welcome to refine this feature.
> > >> >> >> >
> > >> >> >> > Thanks,
> > >> >> >> > Weiqing
> > >> >> >> >
> > >> >> >>
> > >> >>
> > >>
> >
>
Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Thanks for reviewing, Xuyang!
Xingcan (@xingcanc@gmail.com) – do you have any concerns?
If no further objections arise from anyone, I’ll proceed to mark FLIP as
ready for voting.
Best regards,
Weiqing
On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xy...@163.com> wrote:
> LGTM overall. Thanks for updating. I have no problem and +1 for this
> feature.
>
>
>
>
> --
>
> Best！
> Xuyang
>
>
>
>
>
> 在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com> 写道：
> >Hi Xuyang,
> >
> >Thank you for your detailed feedback! I’ve updated the proposal doc
> >accordingly. Please feel free to take another look and let me know if you
> >have any further thoughts or suggestions.
> >
> >Best regards,
> >Weiqing
> >
> >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
> >
> >> Hi, Weiqing.
> >>
> >> After reading the new FLIP, I have no issues with the part `public
> >> interface`. I only have some questions regarding
> >>
> >> the details in the Proposed Changes section.
> >>
> >> Regarding the ModifyKind and UpdateKind of the IntervalJoin node, IIUC:
> >>
> >> - When early firing is enabled, the UpdateKind of the IntervalJoin can
> be
> >> either ONLY_UPDATE_AFTER or
> >>
> >> degrade to BEFORE_AND_AFTER, depending entirely on the requirements of
> the
> >> sink. And the ModifyKind is always ALL.
> >>
> >> - When early firing is disabled, the UpdateKind of the IntervalJoin is
> >> NONE, and the ModifyKind is INSERT.
> >>
> >> - Nevertheless, whether early firing is enabled or disabled, the
> >> IntervalJoin should always require its input to keep
> >>
> >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Best！
> >> Xuyang
> >>
> >>
> >>
> >>
> >>
> >> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com>
> wrote:
> >> >Hi Xingcan and Xuyang,
> >> >
> >> >Thanks so much for the feedback - it was very helpful!
> >> >
> >> >*> 1. The current output stream of a time interval outer join is an
> >> >append-only stream. This change will make it a potential retractable
> >> >stream. I'm not sure if the planner supports a dynamic output type like
> >> >that. Could you add this part to your design doc?*
> >> >
> >> >
> >> > - Yes, enabling early firing on time interval outer joins can emit
> >> > retractions when previously emitted rows are updated or invalidated
> by
> >> > later matches. I’ve updated the proposal (Planner Awareness
> >> > <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.y5w17oloacws
> >> >
> >> > and Changes in FlinkChangelogModeInferenceProgram
> >> > <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
> >> >)
> >> > to clarify that the stream might switch from append-only to a
> >> > retract/upsert stream. Let me know if anything is missing.
> >> >
> >> >
> >> >*> 2. What's the use case when the downstream components need to get
> the
> >> >early fired results regularly?*
> >> >
> >> >
> >> > - The new INTERVAL option (in addition to DELAY) allows periodic
> >> updates
> >> > (e.g., every 10 minutes) after the initial delay. This captures how
> >> results
> >> > evolve over time, similar to Apache Beam’s “Repeatedly” option.
> >> >
> >> >
> >> >*> 3. The time interval join operator itself is not quite efficient
> when
> >> >the state becomes large. Will there be any extra overhead after
> >> introducing
> >> >this feature?*
> >> >
> >> > - Early fire does introduce some overhead by potentially emitting
> >> > partial matches multiple times with retraction (avoiding duplicate
> >> outputs
> >> > though). However, if it’s disabled, there is no additional cost.
> Most
> >> users
> >> > find the performance trade-off acceptable for the real-time
> insights it
> >> > provides.
> >> >
> >> >
> >> >*> 1. Currently, there are some configs related to early firing
> available
> >> >to users: `table.exec.emit.early-fire.en**abled` and
> >> >`table.exec.emit.early-fire.de <http://table.exec.emit.early-fire.de
> >> >**lay`.
> >> >Although their documentation states that they are only applicable to
> the
> >> >Window operator, it seems possible that they may also be relevant in
> the
> >> >context of this FLIP. Otherwise, having different early firing
> behaviors
> >> >for different operators could confuse users.*
> >> >
> >> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve
> added a
> >> > section
> >> > <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q
> >> >
> >> >in
> >> > the proposal highlighting that we should align hint-based interval
> join
> >> > configurations with the existing table.exec.emit.* settings.
> >> Suggestions
> >> > on how to make the unification are welcome! We plan to extend early
> >> firing
> >> > to window joins via hints in a future FLIP.
> >> >
> >> >
> >> >*> 2. The design of `time_mode` is excellent. Similar to point 1,
> perhaps
> >> >we can introduce it to other window-related operators in the future.>
> 3.
> >> >You need to modify the FlinkChangelogModeInferenceProgram to ensure
> that
> >> >downstream nodes of interval joins with early firing enabled are aware
> of
> >> >retract or upsert messages.*
> >> >
> >> > - We agree that time_mode could be introduced to other window-based
> >> > operators down the road. We also want to support early fire for
> >> > window join. Also, thanks for highlighting
> >> > FlinkChangelogModeInferenceProgram! I added the code change on it
> >> > <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
> >> >
> >> > in the proposal.
> >> >
> >> >
> >> >Thanks again for your time and feedback! I’ve updated the proposal with
> >> >these points. Please let me know if there’s anything else I should
> >> address.
> >> >
> >> >Best,
> >> >Weiqing
> >> >
> >> >
> >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
> >> >
> >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few
> questions:
> >> >>
> >> >> 1. Currently, there are some configs related to early firing
> available
> >> to
> >> >> users: `table.exec.emit.early-fire.enabled` and
> >> >>
> >> >> `table.exec.emit.early-fire.delay`. Although their documentation
> states
> >> >> that they are only applicable to the Window operator,
> >> >>
> >> >> it seems possible that they may also be relevant in the context of
> this
> >> >> FLIP. Otherwise, having different early firing behaviors
> >> >>
> >> >> for different operators could confuse users.
> >> >>
> >> >> 2. The design of `time_mode` is excellent. Similar to point 1,
> perhaps
> >> we
> >> >> can introduce it to other window-related operators
> >> >>
> >> >> in the future.
> >> >>
> >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to
> ensure
> >> >> that downstream nodes of interval joins with
> >> >>
> >> >> early firing enabled are aware of retract or upsert messages.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Best！
> >> >> Xuyang
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
> >> >> >Hi Weiqing,
> >> >> >
> >> >> >Thanks for the proposal. IMO, adding early fire for time interval
> outer
> >> >> >joins is feasible overall. I have a few questions.
> >> >> >
> >> >> >1. The current output stream of a time interval outer join is an
> >> >> >append-only stream. This change will make it a potential retractable
> >> >> >stream. I'm not sure if the planner supports a dynamic output type
> like
> >> >> >that. Could you add this part to your design doc?
> >> >> >2. What's the use case when the downstream components need to get
> the
> >> >> early
> >> >> >fired results regularly?
> >> >> >3. The time interval join operator itself is not quite efficient
> when
> >> the
> >> >> >state becomes large. Will there be any extra overhead after
> introducing
> >> >> >this feature?
> >> >> >
> >> >> >Thanks,
> >> >> >Xingcan
> >> >> >
> >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> yangweiqing001@gmail.com>
> >> >> >wrote:
> >> >> >
> >> >> >> Hi all,
> >> >> >>
> >> >> >> Just a gentle reminder regarding the proposal I shared on early
> fire
> >> >> >> support for Flink SQL interval joins. I’d greatly appreciate your
> >> >> feedback
> >> >> >> or suggestions.
> >> >> >>
> >> >> >> Here’s the link to the proposal document: Link
> >> >> >> <
> >> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >> >> >
> >> >> >>
> >> >> >> Thank you!
> >> >> >>
> >> >> >> Best,
> >> >> >> Weiqing
> >> >> >>
> >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> >> yangweiqing001@gmail.com
> >> >> >
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Hi all,
> >> >> >> >
> >> >> >> > I’d like to initiate a discussion about introducing early fire
> >> support
> >> >> >> for
> >> >> >> > Flink SQL interval joins.
> >> >> >> >
> >> >> >> > *Motivation*
> >> >> >> > In many streaming applications, particularly real-time analytics
> >> and
> >> >> >> > monitoring systems, it is valuable to obtain partial results
> >> earlier
> >> >> >> rather
> >> >> >> > than waiting for full join conditions to finalize. For Flink SQL
> >> >> interval
> >> >> >> > joins, results are typically delayed until watermarks ensure no
> >> more
> >> >> >> > matches can occur. This delay can be challenging for scenarios
> that
> >> >> >> require
> >> >> >> > fast feedback. Early fire support addresses this by emitting
> >> >> intermediate
> >> >> >> > results speculatively and using retractions or updates to
> maintain
> >> >> >> eventual
> >> >> >> > consistency and ensure correctness.
> >> >> >> >
> >> >> >> > Here’s the proposal document: Link
> >> >> >> > <
> >> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >> >> >
> >> >> >> >
> >> >> >> > Your feedback and ideas are welcome to refine this feature.
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Weiqing
> >> >> >> >
> >> >> >>
> >> >>
> >>
>
Re:Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Xuyang <xy...@163.com>.
LGTM overall. Thanks for updating. I have no problem and +1 for this feature.
--
Best！
Xuyang
在 2025-01-15 12:33:16，"Weiqing Yang" <ya...@gmail.com> 写道：
>Hi Xuyang,
>
>Thank you for your detailed feedback! I’ve updated the proposal doc
>accordingly. Please feel free to take another look and let me know if you
>have any further thoughts or suggestions.
>
>Best regards,
>Weiqing
>
>On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
>
>> Hi, Weiqing.
>>
>> After reading the new FLIP, I have no issues with the part `public
>> interface`. I only have some questions regarding
>>
>> the details in the Proposed Changes section.
>>
>> Regarding the ModifyKind and UpdateKind of the IntervalJoin node, IIUC:
>>
>> - When early firing is enabled, the UpdateKind of the IntervalJoin can be
>> either ONLY_UPDATE_AFTER or
>>
>> degrade to BEFORE_AND_AFTER, depending entirely on the requirements of the
>> sink. And the ModifyKind is always ALL.
>>
>> - When early firing is disabled, the UpdateKind of the IntervalJoin is
>> NONE, and the ModifyKind is INSERT.
>>
>> - Nevertheless, whether early firing is enabled or disabled, the
>> IntervalJoin should always require its input to keep
>>
>> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
>>
>>
>>
>>
>> --
>>
>> Best！
>> Xuyang
>>
>>
>>
>>
>>
>> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com> wrote:
>> >Hi Xingcan and Xuyang,
>> >
>> >Thanks so much for the feedback - it was very helpful!
>> >
>> >*> 1. The current output stream of a time interval outer join is an
>> >append-only stream. This change will make it a potential retractable
>> >stream. I'm not sure if the planner supports a dynamic output type like
>> >that. Could you add this part to your design doc?*
>> >
>> >
>> > - Yes, enabling early firing on time interval outer joins can emit
>> > retractions when previously emitted rows are updated or invalidated by
>> > later matches. I’ve updated the proposal (Planner Awareness
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.y5w17oloacws
>> >
>> > and Changes in FlinkChangelogModeInferenceProgram
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
>> >)
>> > to clarify that the stream might switch from append-only to a
>> > retract/upsert stream. Let me know if anything is missing.
>> >
>> >
>> >*> 2. What's the use case when the downstream components need to get the
>> >early fired results regularly?*
>> >
>> >
>> > - The new INTERVAL option (in addition to DELAY) allows periodic
>> updates
>> > (e.g., every 10 minutes) after the initial delay. This captures how
>> results
>> > evolve over time, similar to Apache Beam’s “Repeatedly” option.
>> >
>> >
>> >*> 3. The time interval join operator itself is not quite efficient when
>> >the state becomes large. Will there be any extra overhead after
>> introducing
>> >this feature?*
>> >
>> > - Early fire does introduce some overhead by potentially emitting
>> > partial matches multiple times with retraction (avoiding duplicate
>> outputs
>> > though). However, if it’s disabled, there is no additional cost. Most
>> users
>> > find the performance trade-off acceptable for the real-time insights it
>> > provides.
>> >
>> >
>> >*> 1. Currently, there are some configs related to early firing available
>> >to users: `table.exec.emit.early-fire.en**abled` and
>> >`table.exec.emit.early-fire.de <http://table.exec.emit.early-fire.de
>> >**lay`.
>> >Although their documentation states that they are only applicable to the
>> >Window operator, it seems possible that they may also be relevant in the
>> >context of this FLIP. Otherwise, having different early firing behaviors
>> >for different operators could confuse users.*
>> >
>> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve added a
>> > section
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q
>> >
>> >in
>> > the proposal highlighting that we should align hint-based interval join
>> > configurations with the existing table.exec.emit.* settings.
>> Suggestions
>> > on how to make the unification are welcome! We plan to extend early
>> firing
>> > to window joins via hints in a future FLIP.
>> >
>> >
>> >*> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
>> >we can introduce it to other window-related operators in the future.> 3.
>> >You need to modify the FlinkChangelogModeInferenceProgram to ensure that
>> >downstream nodes of interval joins with early firing enabled are aware of
>> >retract or upsert messages.*
>> >
>> > - We agree that time_mode could be introduced to other window-based
>> > operators down the road. We also want to support early fire for
>> > window join. Also, thanks for highlighting
>> > FlinkChangelogModeInferenceProgram! I added the code change on it
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
>> >
>> > in the proposal.
>> >
>> >
>> >Thanks again for your time and feedback! I’ve updated the proposal with
>> >these points. Please let me know if there’s anything else I should
>> address.
>> >
>> >Best,
>> >Weiqing
>> >
>> >
>> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
>> >
>> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
>> >>
>> >> 1. Currently, there are some configs related to early firing available
>> to
>> >> users: `table.exec.emit.early-fire.enabled` and
>> >>
>> >> `table.exec.emit.early-fire.delay`. Although their documentation states
>> >> that they are only applicable to the Window operator,
>> >>
>> >> it seems possible that they may also be relevant in the context of this
>> >> FLIP. Otherwise, having different early firing behaviors
>> >>
>> >> for different operators could confuse users.
>> >>
>> >> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
>> we
>> >> can introduce it to other window-related operators
>> >>
>> >> in the future.
>> >>
>> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to ensure
>> >> that downstream nodes of interval joins with
>> >>
>> >> early firing enabled are aware of retract or upsert messages.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Best！
>> >> Xuyang
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
>> >> >Hi Weiqing,
>> >> >
>> >> >Thanks for the proposal. IMO, adding early fire for time interval outer
>> >> >joins is feasible overall. I have a few questions.
>> >> >
>> >> >1. The current output stream of a time interval outer join is an
>> >> >append-only stream. This change will make it a potential retractable
>> >> >stream. I'm not sure if the planner supports a dynamic output type like
>> >> >that. Could you add this part to your design doc?
>> >> >2. What's the use case when the downstream components need to get the
>> >> early
>> >> >fired results regularly?
>> >> >3. The time interval join operator itself is not quite efficient when
>> the
>> >> >state becomes large. Will there be any extra overhead after introducing
>> >> >this feature?
>> >> >
>> >> >Thanks,
>> >> >Xingcan
>> >> >
>> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
>> >> >wrote:
>> >> >
>> >> >> Hi all,
>> >> >>
>> >> >> Just a gentle reminder regarding the proposal I shared on early fire
>> >> >> support for Flink SQL interval joins. I’d greatly appreciate your
>> >> feedback
>> >> >> or suggestions.
>> >> >>
>> >> >> Here’s the link to the proposal document: Link
>> >> >> <
>> >> >>
>> >>
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >> >> >
>> >> >>
>> >> >> Thank you!
>> >> >>
>> >> >> Best,
>> >> >> Weiqing
>> >> >>
>> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
>> yangweiqing001@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >> > Hi all,
>> >> >> >
>> >> >> > I’d like to initiate a discussion about introducing early fire
>> support
>> >> >> for
>> >> >> > Flink SQL interval joins.
>> >> >> >
>> >> >> > *Motivation*
>> >> >> > In many streaming applications, particularly real-time analytics
>> and
>> >> >> > monitoring systems, it is valuable to obtain partial results
>> earlier
>> >> >> rather
>> >> >> > than waiting for full join conditions to finalize. For Flink SQL
>> >> interval
>> >> >> > joins, results are typically delayed until watermarks ensure no
>> more
>> >> >> > matches can occur. This delay can be challenging for scenarios that
>> >> >> require
>> >> >> > fast feedback. Early fire support addresses this by emitting
>> >> intermediate
>> >> >> > results speculatively and using retractions or updates to maintain
>> >> >> eventual
>> >> >> > consistency and ensure correctness.
>> >> >> >
>> >> >> > Here’s the proposal document: Link
>> >> >> > <
>> >> >>
>> >>
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >> >> >
>> >> >> >
>> >> >> > Your feedback and ideas are welcome to refine this feature.
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Weiqing
>> >> >> >
>> >> >>
>> >>
>>
Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Xuyang,
Thank you for your detailed feedback! I’ve updated the proposal doc
accordingly. Please feel free to take another look and let me know if you
have any further thoughts or suggestions.
Best regards,
Weiqing
On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xy...@163.com> wrote:
> Hi, Weiqing.
>
> After reading the new FLIP, I have no issues with the part `public
> interface`. I only have some questions regarding
>
> the details in the Proposed Changes section.
>
> Regarding the ModifyKind and UpdateKind of the IntervalJoin node, IIUC:
>
> - When early firing is enabled, the UpdateKind of the IntervalJoin can be
> either ONLY_UPDATE_AFTER or
>
> degrade to BEFORE_AND_AFTER, depending entirely on the requirements of the
> sink. And the ModifyKind is always ALL.
>
> - When early firing is disabled, the UpdateKind of the IntervalJoin is
> NONE, and the ModifyKind is INSERT.
>
> - Nevertheless, whether early firing is enabled or disabled, the
> IntervalJoin should always require its input to keep
>
> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
>
>
>
>
> --
>
> Best！
> Xuyang
>
>
>
>
>
> At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com> wrote:
> >Hi Xingcan and Xuyang,
> >
> >Thanks so much for the feedback - it was very helpful!
> >
> >*> 1. The current output stream of a time interval outer join is an
> >append-only stream. This change will make it a potential retractable
> >stream. I'm not sure if the planner supports a dynamic output type like
> >that. Could you add this part to your design doc?*
> >
> >
> > - Yes, enabling early firing on time interval outer joins can emit
> > retractions when previously emitted rows are updated or invalidated by
> > later matches. I’ve updated the proposal (Planner Awareness
> > <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.y5w17oloacws
> >
> > and Changes in FlinkChangelogModeInferenceProgram
> > <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
> >)
> > to clarify that the stream might switch from append-only to a
> > retract/upsert stream. Let me know if anything is missing.
> >
> >
> >*> 2. What's the use case when the downstream components need to get the
> >early fired results regularly?*
> >
> >
> > - The new INTERVAL option (in addition to DELAY) allows periodic
> updates
> > (e.g., every 10 minutes) after the initial delay. This captures how
> results
> > evolve over time, similar to Apache Beam’s “Repeatedly” option.
> >
> >
> >*> 3. The time interval join operator itself is not quite efficient when
> >the state becomes large. Will there be any extra overhead after
> introducing
> >this feature?*
> >
> > - Early fire does introduce some overhead by potentially emitting
> > partial matches multiple times with retraction (avoiding duplicate
> outputs
> > though). However, if it’s disabled, there is no additional cost. Most
> users
> > find the performance trade-off acceptable for the real-time insights it
> > provides.
> >
> >
> >*> 1. Currently, there are some configs related to early firing available
> >to users: `table.exec.emit.early-fire.en**abled` and
> >`table.exec.emit.early-fire.de <http://table.exec.emit.early-fire.de
> >**lay`.
> >Although their documentation states that they are only applicable to the
> >Window operator, it seems possible that they may also be relevant in the
> >context of this FLIP. Otherwise, having different early firing behaviors
> >for different operators could confuse users.*
> >
> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve added a
> > section
> > <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q
> >
> >in
> > the proposal highlighting that we should align hint-based interval join
> > configurations with the existing table.exec.emit.* settings.
> Suggestions
> > on how to make the unification are welcome! We plan to extend early
> firing
> > to window joins via hints in a future FLIP.
> >
> >
> >*> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
> >we can introduce it to other window-related operators in the future.> 3.
> >You need to modify the FlinkChangelogModeInferenceProgram to ensure that
> >downstream nodes of interval joins with early firing enabled are aware of
> >retract or upsert messages.*
> >
> > - We agree that time_mode could be introduced to other window-based
> > operators down the road. We also want to support early fire for
> > window join. Also, thanks for highlighting
> > FlinkChangelogModeInferenceProgram! I added the code change on it
> > <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u
> >
> > in the proposal.
> >
> >
> >Thanks again for your time and feedback! I’ve updated the proposal with
> >these points. Please let me know if there’s anything else I should
> address.
> >
> >Best,
> >Weiqing
> >
> >
> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
> >
> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
> >>
> >> 1. Currently, there are some configs related to early firing available
> to
> >> users: `table.exec.emit.early-fire.enabled` and
> >>
> >> `table.exec.emit.early-fire.delay`. Although their documentation states
> >> that they are only applicable to the Window operator,
> >>
> >> it seems possible that they may also be relevant in the context of this
> >> FLIP. Otherwise, having different early firing behaviors
> >>
> >> for different operators could confuse users.
> >>
> >> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
> we
> >> can introduce it to other window-related operators
> >>
> >> in the future.
> >>
> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to ensure
> >> that downstream nodes of interval joins with
> >>
> >> early firing enabled are aware of retract or upsert messages.
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Best！
> >> Xuyang
> >>
> >>
> >>
> >>
> >>
> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
> >> >Hi Weiqing,
> >> >
> >> >Thanks for the proposal. IMO, adding early fire for time interval outer
> >> >joins is feasible overall. I have a few questions.
> >> >
> >> >1. The current output stream of a time interval outer join is an
> >> >append-only stream. This change will make it a potential retractable
> >> >stream. I'm not sure if the planner supports a dynamic output type like
> >> >that. Could you add this part to your design doc?
> >> >2. What's the use case when the downstream components need to get the
> >> early
> >> >fired results regularly?
> >> >3. The time interval join operator itself is not quite efficient when
> the
> >> >state becomes large. Will there be any extra overhead after introducing
> >> >this feature?
> >> >
> >> >Thanks,
> >> >Xingcan
> >> >
> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
> >> >wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> Just a gentle reminder regarding the proposal I shared on early fire
> >> >> support for Flink SQL interval joins. I’d greatly appreciate your
> >> feedback
> >> >> or suggestions.
> >> >>
> >> >> Here’s the link to the proposal document: Link
> >> >> <
> >> >>
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >> >
> >> >>
> >> >> Thank you!
> >> >>
> >> >> Best,
> >> >> Weiqing
> >> >>
> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> yangweiqing001@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >> > Hi all,
> >> >> >
> >> >> > I’d like to initiate a discussion about introducing early fire
> support
> >> >> for
> >> >> > Flink SQL interval joins.
> >> >> >
> >> >> > *Motivation*
> >> >> > In many streaming applications, particularly real-time analytics
> and
> >> >> > monitoring systems, it is valuable to obtain partial results
> earlier
> >> >> rather
> >> >> > than waiting for full join conditions to finalize. For Flink SQL
> >> interval
> >> >> > joins, results are typically delayed until watermarks ensure no
> more
> >> >> > matches can occur. This delay can be challenging for scenarios that
> >> >> require
> >> >> > fast feedback. Early fire support addresses this by emitting
> >> intermediate
> >> >> > results speculatively and using retractions or updates to maintain
> >> >> eventual
> >> >> > consistency and ensure correctness.
> >> >> >
> >> >> > Here’s the proposal document: Link
> >> >> > <
> >> >>
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >> >
> >> >> >
> >> >> > Your feedback and ideas are welcome to refine this feature.
> >> >> >
> >> >> > Thanks,
> >> >> > Weiqing
> >> >> >
> >> >>
> >>
>
Re:Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Xuyang <xy...@163.com>.
Hi, Weiqing.
After reading the new FLIP, I have no issues with the part `public interface`. I only have some questions regarding
the details in the Proposed Changes section.
Regarding the ModifyKind and UpdateKind of the IntervalJoin node, IIUC:
- When early firing is enabled, the UpdateKind of the IntervalJoin can be either ONLY_UPDATE_AFTER or
degrade to BEFORE_AND_AFTER, depending entirely on the requirements of the sink. And the ModifyKind is always ALL.
- When early firing is disabled, the UpdateKind of the IntervalJoin is NONE, and the ModifyKind is INSERT.
- Nevertheless, whether early firing is enabled or disabled, the IntervalJoin should always require its input to keep
ModifyKind with INSERT_ONLY and UpdateKind with NONE.
--
Best！
Xuyang
At 2025-01-09 15:30:44, "Weiqing Yang" <ya...@gmail.com> wrote:
>Hi Xingcan and Xuyang,
>
>Thanks so much for the feedback - it was very helpful!
>
>*> 1. The current output stream of a time interval outer join is an
>append-only stream. This change will make it a potential retractable
>stream. I'm not sure if the planner supports a dynamic output type like
>that. Could you add this part to your design doc?*
>
>
> - Yes, enabling early firing on time interval outer joins can emit
> retractions when previously emitted rows are updated or invalidated by
> later matches. I’ve updated the proposal (Planner Awareness
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.y5w17oloacws>
> and Changes in FlinkChangelogModeInferenceProgram
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u>)
> to clarify that the stream might switch from append-only to a
> retract/upsert stream. Let me know if anything is missing.
>
>
>*> 2. What's the use case when the downstream components need to get the
>early fired results regularly?*
>
>
> - The new INTERVAL option (in addition to DELAY) allows periodic updates
> (e.g., every 10 minutes) after the initial delay. This captures how results
> evolve over time, similar to Apache Beam’s “Repeatedly” option.
>
>
>*> 3. The time interval join operator itself is not quite efficient when
>the state becomes large. Will there be any extra overhead after introducing
>this feature?*
>
> - Early fire does introduce some overhead by potentially emitting
> partial matches multiple times with retraction (avoiding duplicate outputs
> though). However, if it’s disabled, there is no additional cost. Most users
> find the performance trade-off acceptable for the real-time insights it
> provides.
>
>
>*> 1. Currently, there are some configs related to early firing available
>to users: `table.exec.emit.early-fire.en**abled` and
>`table.exec.emit.early-fire.de <http://table.exec.emit.early-fire.de>**lay`.
>Although their documentation states that they are only applicable to the
>Window operator, it seems possible that they may also be relevant in the
>context of this FLIP. Otherwise, having different early firing behaviors
>for different operators could confuse users.*
>
> - +1 on unifying early-fire behaviors to avoid confusion. I’ve added a
> section
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q>
>in
> the proposal highlighting that we should align hint-based interval join
> configurations with the existing table.exec.emit.* settings. Suggestions
> on how to make the unification are welcome! We plan to extend early firing
> to window joins via hints in a future FLIP.
>
>
>*> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
>we can introduce it to other window-related operators in the future.> 3.
>You need to modify the FlinkChangelogModeInferenceProgram to ensure that
>downstream nodes of interval joins with early firing enabled are aware of
>retract or upsert messages.*
>
> - We agree that time_mode could be introduced to other window-based
> operators down the road. We also want to support early fire for
> window join. Also, thanks for highlighting
> FlinkChangelogModeInferenceProgram! I added the code change on it
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u>
> in the proposal.
>
>
>Thanks again for your time and feedback! I’ve updated the proposal with
>these points. Please let me know if there’s anything else I should address.
>
>Best,
>Weiqing
>
>
>On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
>
>> Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
>>
>> 1. Currently, there are some configs related to early firing available to
>> users: `table.exec.emit.early-fire.enabled` and
>>
>> `table.exec.emit.early-fire.delay`. Although their documentation states
>> that they are only applicable to the Window operator,
>>
>> it seems possible that they may also be relevant in the context of this
>> FLIP. Otherwise, having different early firing behaviors
>>
>> for different operators could confuse users.
>>
>> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps we
>> can introduce it to other window-related operators
>>
>> in the future.
>>
>> 3. You need to modify the FlinkChangelogModeInferenceProgram to ensure
>> that downstream nodes of interval joins with
>>
>> early firing enabled are aware of retract or upsert messages.
>>
>>
>>
>>
>> --
>>
>> Best！
>> Xuyang
>>
>>
>>
>>
>>
>> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
>> >Hi Weiqing,
>> >
>> >Thanks for the proposal. IMO, adding early fire for time interval outer
>> >joins is feasible overall. I have a few questions.
>> >
>> >1. The current output stream of a time interval outer join is an
>> >append-only stream. This change will make it a potential retractable
>> >stream. I'm not sure if the planner supports a dynamic output type like
>> >that. Could you add this part to your design doc?
>> >2. What's the use case when the downstream components need to get the
>> early
>> >fired results regularly?
>> >3. The time interval join operator itself is not quite efficient when the
>> >state becomes large. Will there be any extra overhead after introducing
>> >this feature?
>> >
>> >Thanks,
>> >Xingcan
>> >
>> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
>> >wrote:
>> >
>> >> Hi all,
>> >>
>> >> Just a gentle reminder regarding the proposal I shared on early fire
>> >> support for Flink SQL interval joins. I’d greatly appreciate your
>> feedback
>> >> or suggestions.
>> >>
>> >> Here’s the link to the proposal document: Link
>> >> <
>> >>
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >> >
>> >>
>> >> Thank you!
>> >>
>> >> Best,
>> >> Weiqing
>> >>
>> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <yangweiqing001@gmail.com
>> >
>> >> wrote:
>> >>
>> >> > Hi all,
>> >> >
>> >> > I’d like to initiate a discussion about introducing early fire support
>> >> for
>> >> > Flink SQL interval joins.
>> >> >
>> >> > *Motivation*
>> >> > In many streaming applications, particularly real-time analytics and
>> >> > monitoring systems, it is valuable to obtain partial results earlier
>> >> rather
>> >> > than waiting for full join conditions to finalize. For Flink SQL
>> interval
>> >> > joins, results are typically delayed until watermarks ensure no more
>> >> > matches can occur. This delay can be challenging for scenarios that
>> >> require
>> >> > fast feedback. Early fire support addresses this by emitting
>> intermediate
>> >> > results speculatively and using retractions or updates to maintain
>> >> eventual
>> >> > consistency and ensure correctness.
>> >> >
>> >> > Here’s the proposal document: Link
>> >> > <
>> >>
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >> >
>> >> >
>> >> > Your feedback and ideas are welcome to refine this feature.
>> >> >
>> >> > Thanks,
>> >> > Weiqing
>> >> >
>> >>
>>
Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Xingcan and Xuyang,
Thanks so much for the feedback - it was very helpful!
*> 1. The current output stream of a time interval outer join is an
append-only stream. This change will make it a potential retractable
stream. I'm not sure if the planner supports a dynamic output type like
that. Could you add this part to your design doc?*
- Yes, enabling early firing on time interval outer joins can emit
retractions when previously emitted rows are updated or invalidated by
later matches. I’ve updated the proposal (Planner Awareness
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.y5w17oloacws>
and Changes in FlinkChangelogModeInferenceProgram
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u>)
to clarify that the stream might switch from append-only to a
retract/upsert stream. Let me know if anything is missing.
*> 2. What's the use case when the downstream components need to get the
early fired results regularly?*
- The new INTERVAL option (in addition to DELAY) allows periodic updates
(e.g., every 10 minutes) after the initial delay. This captures how results
evolve over time, similar to Apache Beam’s “Repeatedly” option.
*> 3. The time interval join operator itself is not quite efficient when
the state becomes large. Will there be any extra overhead after introducing
this feature?*
- Early fire does introduce some overhead by potentially emitting
partial matches multiple times with retraction (avoiding duplicate outputs
though). However, if it’s disabled, there is no additional cost. Most users
find the performance trade-off acceptable for the real-time insights it
provides.
*> 1. Currently, there are some configs related to early firing available
to users: `table.exec.emit.early-fire.en**abled` and
`table.exec.emit.early-fire.de <http://table.exec.emit.early-fire.de>**lay`.
Although their documentation states that they are only applicable to the
Window operator, it seems possible that they may also be relevant in the
context of this FLIP. Otherwise, having different early firing behaviors
for different operators could confuse users.*
- +1 on unifying early-fire behaviors to avoid confusion. I’ve added a
section
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.rr0i3gmdjt4q>
in
the proposal highlighting that we should align hint-based interval join
configurations with the existing table.exec.emit.* settings. Suggestions
on how to make the unification are welcome! We plan to extend early firing
to window joins via hints in a future FLIP.
*> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps
we can introduce it to other window-related operators in the future.> 3.
You need to modify the FlinkChangelogModeInferenceProgram to ensure that
downstream nodes of interval joins with early firing enabled are aware of
retract or upsert messages.*
- We agree that time_mode could be introduced to other window-based
operators down the road. We also want to support early fire for
window join. Also, thanks for highlighting
FlinkChangelogModeInferenceProgram! I added the code change on it
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z6qdwrvtgn4u>
in the proposal.
Thanks again for your time and feedback! I’ve updated the proposal with
these points. Please let me know if there’s anything else I should address.
Best,
Weiqing
On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xy...@163.com> wrote:
> Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
>
> 1. Currently, there are some configs related to early firing available to
> users: `table.exec.emit.early-fire.enabled` and
>
> `table.exec.emit.early-fire.delay`. Although their documentation states
> that they are only applicable to the Window operator,
>
> it seems possible that they may also be relevant in the context of this
> FLIP. Otherwise, having different early firing behaviors
>
> for different operators could confuse users.
>
> 2. The design of `time_mode` is excellent. Similar to point 1, perhaps we
> can introduce it to other window-related operators
>
> in the future.
>
> 3. You need to modify the FlinkChangelogModeInferenceProgram to ensure
> that downstream nodes of interval joins with
>
> early firing enabled are aware of retract or upsert messages.
>
>
>
>
> --
>
> Best！
> Xuyang
>
>
>
>
>
> At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
> >Hi Weiqing,
> >
> >Thanks for the proposal. IMO, adding early fire for time interval outer
> >joins is feasible overall. I have a few questions.
> >
> >1. The current output stream of a time interval outer join is an
> >append-only stream. This change will make it a potential retractable
> >stream. I'm not sure if the planner supports a dynamic output type like
> >that. Could you add this part to your design doc?
> >2. What's the use case when the downstream components need to get the
> early
> >fired results regularly?
> >3. The time interval join operator itself is not quite efficient when the
> >state becomes large. Will there be any extra overhead after introducing
> >this feature?
> >
> >Thanks,
> >Xingcan
> >
> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> Just a gentle reminder regarding the proposal I shared on early fire
> >> support for Flink SQL interval joins. I’d greatly appreciate your
> feedback
> >> or suggestions.
> >>
> >> Here’s the link to the proposal document: Link
> >> <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >
> >>
> >> Thank you!
> >>
> >> Best,
> >> Weiqing
> >>
> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <yangweiqing001@gmail.com
> >
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I’d like to initiate a discussion about introducing early fire support
> >> for
> >> > Flink SQL interval joins.
> >> >
> >> > *Motivation*
> >> > In many streaming applications, particularly real-time analytics and
> >> > monitoring systems, it is valuable to obtain partial results earlier
> >> rather
> >> > than waiting for full join conditions to finalize. For Flink SQL
> interval
> >> > joins, results are typically delayed until watermarks ensure no more
> >> > matches can occur. This delay can be challenging for scenarios that
> >> require
> >> > fast feedback. Early fire support addresses this by emitting
> intermediate
> >> > results speculatively and using retractions or updates to maintain
> >> eventual
> >> > consistency and ensure correctness.
> >> >
> >> > Here’s the proposal document: Link
> >> > <
> >>
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >> >
> >> >
> >> > Your feedback and ideas are welcome to refine this feature.
> >> >
> >> > Thanks,
> >> > Weiqing
> >> >
> >>
>
Query Regarding encryption support in RocksDB
Posted by MUKUL GUPTA <mu...@ericsson.com.INVALID>.
Hello All,
I have a Use case regarding storing sensitive data in flink state which gets regularly checkpointed in RocksDB. At RocksDb path I can see created file is stored in binary format which is easily readable.
Following are my queries:
- Is there any possibility to have RocksDB encrypt/secure the data before storing?
- Are there any RocksDB option for the same?
Kindly let us know.
Thanks and Regards
Mukul Gupta
Re:Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Xuyang <xy...@163.com>.
Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
1. Currently, there are some configs related to early firing available to users: `table.exec.emit.early-fire.enabled` and
`table.exec.emit.early-fire.delay`. Although their documentation states that they are only applicable to the Window operator,
it seems possible that they may also be relevant in the context of this FLIP. Otherwise, having different early firing behaviors
for different operators could confuse users.
2. The design of `time_mode` is excellent. Similar to point 1, perhaps we can introduce it to other window-related operators
in the future.
3. You need to modify the FlinkChangelogModeInferenceProgram to ensure that downstream nodes of interval joins with
early firing enabled are aware of retract or upsert messages.
--
Best！
Xuyang
At 2025-01-07 06:35:51, "Xingcan Cui" <xi...@gmail.com> wrote:
>Hi Weiqing,
>
>Thanks for the proposal. IMO, adding early fire for time interval outer
>joins is feasible overall. I have a few questions.
>
>1. The current output stream of a time interval outer join is an
>append-only stream. This change will make it a potential retractable
>stream. I'm not sure if the planner supports a dynamic output type like
>that. Could you add this part to your design doc?
>2. What's the use case when the downstream components need to get the early
>fired results regularly?
>3. The time interval join operator itself is not quite efficient when the
>state becomes large. Will there be any extra overhead after introducing
>this feature?
>
>Thanks,
>Xingcan
>
>On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> Just a gentle reminder regarding the proposal I shared on early fire
>> support for Flink SQL interval joins. I’d greatly appreciate your feedback
>> or suggestions.
>>
>> Here’s the link to the proposal document: Link
>> <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >
>>
>> Thank you!
>>
>> Best,
>> Weiqing
>>
>> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <ya...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > I’d like to initiate a discussion about introducing early fire support
>> for
>> > Flink SQL interval joins.
>> >
>> > *Motivation*
>> > In many streaming applications, particularly real-time analytics and
>> > monitoring systems, it is valuable to obtain partial results earlier
>> rather
>> > than waiting for full join conditions to finalize. For Flink SQL interval
>> > joins, results are typically delayed until watermarks ensure no more
>> > matches can occur. This delay can be challenging for scenarios that
>> require
>> > fast feedback. Early fire support addresses this by emitting intermediate
>> > results speculatively and using retractions or updates to maintain
>> eventual
>> > consistency and ensure correctness.
>> >
>> > Here’s the proposal document: Link
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >
>> >
>> > Your feedback and ideas are welcome to refine this feature.
>> >
>> > Thanks,
>> > Weiqing
>> >
>>
Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Xingcan Cui <xi...@gmail.com>.
Hi Weiqing,
Thanks for the proposal. IMO, adding early fire for time interval outer
joins is feasible overall. I have a few questions.
1. The current output stream of a time interval outer join is an
append-only stream. This change will make it a potential retractable
stream. I'm not sure if the planner supports a dynamic output type like
that. Could you add this part to your design doc?
2. What's the use case when the downstream components need to get the early
fired results regularly?
3. The time interval join operator itself is not quite efficient when the
state becomes large. Will there be any extra overhead after introducing
this feature?
Thanks,
Xingcan
On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi all,
>
> Just a gentle reminder regarding the proposal I shared on early fire
> support for Flink SQL interval joins. I’d greatly appreciate your feedback
> or suggestions.
>
> Here’s the link to the proposal document: Link
> <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >
>
> Thank you!
>
> Best,
> Weiqing
>
> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I’d like to initiate a discussion about introducing early fire support
> for
> > Flink SQL interval joins.
> >
> > *Motivation*
> > In many streaming applications, particularly real-time analytics and
> > monitoring systems, it is valuable to obtain partial results earlier
> rather
> > than waiting for full join conditions to finalize. For Flink SQL interval
> > joins, results are typically delayed until watermarks ensure no more
> > matches can occur. This delay can be challenging for scenarios that
> require
> > fast feedback. Early fire support addresses this by emitting intermediate
> > results speculatively and using retractions or updates to maintain
> eventual
> > consistency and ensure correctness.
> >
> > Here’s the proposal document: Link
> > <
> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
> >
> >
> > Your feedback and ideas are welcome to refine this feature.
> >
> > Thanks,
> > Weiqing
> >
>
Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join
Posted by Weiqing Yang <ya...@gmail.com>.
Hi all,
Just a gentle reminder regarding the proposal I shared on early fire
support for Flink SQL interval joins. I’d greatly appreciate your feedback
or suggestions.
Here’s the link to the proposal document: Link
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph>
Thank you!
Best,
Weiqing
On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi all,
>
> I’d like to initiate a discussion about introducing early fire support for
> Flink SQL interval joins.
>
> *Motivation*
> In many streaming applications, particularly real-time analytics and
> monitoring systems, it is valuable to obtain partial results earlier rather
> than waiting for full join conditions to finalize. For Flink SQL interval
> joins, results are typically delayed until watermarks ensure no more
> matches can occur. This delay can be challenging for scenarios that require
> fast feedback. Early fire support addresses this by emitting intermediate
> results speculatively and using retractions or updates to maintain eventual
> consistency and ensure correctness.
>
> Here’s the proposal document: Link
> <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph>
>
> Your feedback and ideas are welcome to refine this feature.
>
> Thanks,
> Weiqing
>