You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Aleksandr Savonin <sa...@gmail.com> on 2026/05/27 13:57:27 UTC
[DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Hi everyone,
I’d like to start a discussion about FLIP-583: Expose JobInfo on
Source contexts [1].
Source connectors often need job metadata (job ID, job name) for
authentication, distributed tracing, metrics labeling, and other
usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
The current workarounds on the source side (e.g. parsing the job ID
from metric group variables) are brittle/fragile and undiscoverable.
The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
`SourceReaderContext` and `SplitEnumeratorContext`, following the same
pattern as already established in these interfaces. The runtime
implementations delegate to `StreamingRuntimeContext` (reader side)
and `OperatorCoordinator.Context` (enumerator side).
Looking forward to your feedback.
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
--
Kind regards,
Aleksandr
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Rui Fan <19...@gmail.com>.
Thanks for the update, I have no more questions.
Best,
Rui
On Fri, Jun 5, 2026 at 3:09 PM Aleksandr Savonin <sa...@gmail.com>
wrote:
> Hi Rui.
> Thank you for the feedback.
> I updated my Umbrella ticket and now it has subtasks.
> Could you please clarify whether you have other questions related to the
> FLIP?
>
> On Tue, 2 Jun 2026 at 19:08, Aleksandr Savonin <sa...@gmail.com>
> wrote:
> >
> > Hi.
> > Thanks Yuepeng Pan and Rui for the feedback and participation!
> >
> > > RichSourceReaderContext is well past that, so its @Experimental tag is
> a stalled promotion, not a genuine lack of stability contract.
> >
> > Agree, I updated my FLIP (Motivation section) accordingly.
> >
> >
> > > Is any of them a real connector today that's actually working around
> this?
> >
> > I've checked some of them, but could not find a connector working
> > around this today, likely because the JobID is unavailable without
> > "hacks" at this moment.
> >
> >
> > > And is there any follow-up JIRA on the connector side to adopt the new
> interface?
> >
> > I've created a follow-up umbrella ticket
> > https://issues.apache.org/jira/browse/FLINK-39827 . I have plans to
> > adopt the proposed API in multiple places/different repos. Happy to
> > elaborate if it makes sense at this moment.
> >
> > On Tue, 2 Jun 2026 at 18:00, Rui Fan <19...@gmail.com> wrote:
> > >
> > > Hey Aleksandr,
> > >
> > > Thanks for driving this — exposing JobInfo to the source context makes
> > > sense, and it's already there on the sink side[1].
> > >
> > > A few questions:
> > > 1. On the "no stability contract" argument for RichSourceReaderContext:
> > >
> > > Per FLIP-321[2], an @Experimental API should be promoted after two
> > > releases. RichSourceReaderContext is well past that, so its
> @Experimental
> > > tag
> > > is a stalled promotion, not a genuine lack of stability contract. The
> real
> > > gaps are
> > > that it only exists on the reader side, and that access goes through
> the
> > > whole
> > > RuntimeContext. Worth reframing the motivation around those instead.
> > >
> > > 2. On the use cases:
> > >
> > > Is any of them a real connector today that's actually working around
> this?
> > > And is there any follow-up JIRA on the connector side to adopt the new
> > > interface?
> > >
> > > Best,
> > > Rui
> > >
> > > [1]
> > >
> https://github.com/apache/flink/blob/a5a7925415bdf6f257b100cc27e9aeb19d096d04/flink-core/src/main/java/org/apache/flink/api/connector/sink2/InitContext.java#L52
> > >
> > > [2]
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-321:+Introduce+an+API+deprecation+process
> > >
> > >
> > > On Tue, Jun 2, 2026 at 4:55 PM Yuepeng Pan <hi...@gmail.com>
> wrote:
> > >
> > > > Hi Aleksandr Savonin,
> > > >
> > > > +1. That sounds reasonable to me.
> > > >
> > > >
> > > > Best,
> > > > Yuepeng Pan
> > > >
> > > > Aleksandr Savonin <sa...@gmail.com> 于2026年6月2日周二 22:21写道：
> > > >
> > > > > Hi everyone,
> > > > > gently pinging this thread to see if anyone has thoughts or
> concerns
> > > > > regarding this proposal.
> > > > >
> > > > > On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <
> savonitar@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I’d like to start a discussion about FLIP-583: Expose JobInfo on
> > > > > > Source contexts [1].
> > > > > > Source connectors often need job metadata (job ID, job name) for
> > > > > > authentication, distributed tracing, metrics labeling, and other
> > > > > > usages. SinkV2 already exposes this via
> `InitContext.getJobInfo()`.
> > > > > >
> > > > > > The current workarounds on the source side (e.g. parsing the job
> ID
> > > > > > from metric group variables) are brittle/fragile and
> undiscoverable.
> > > > > > The FLIP adds a `@PublicEvolving` default method `getJobInfo()`
> to
> > > > > > `SourceReaderContext` and `SplitEnumeratorContext`, following
> the same
> > > > > > pattern as already established in these interfaces. The runtime
> > > > > > implementations delegate to `StreamingRuntimeContext` (reader
> side)
> > > > > > and `OperatorCoordinator.Context` (enumerator side).
> > > > > >
> > > > > > Looking forward to your feedback.
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
> > > > > >
> > > > > > --
> > > > > > Kind regards,
> > > > > > Aleksandr
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kind regards,
> > > > > Aleksandr
> > > > >
> > > >
> >
> >
> >
> > --
> > Kind regards,
> > Aleksandr
>
>
>
> --
> Kind regards,
> Aleksandr
>
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Aleksandr Savonin <sa...@gmail.com>.
Hi Rui.
Thank you for the feedback.
I updated my Umbrella ticket and now it has subtasks.
Could you please clarify whether you have other questions related to the FLIP?
On Tue, 2 Jun 2026 at 19:08, Aleksandr Savonin <sa...@gmail.com> wrote:
>
> Hi.
> Thanks Yuepeng Pan and Rui for the feedback and participation!
>
> > RichSourceReaderContext is well past that, so its @Experimental tag is a stalled promotion, not a genuine lack of stability contract.
>
> Agree, I updated my FLIP (Motivation section) accordingly.
>
>
> > Is any of them a real connector today that's actually working around this?
>
> I've checked some of them, but could not find a connector working
> around this today, likely because the JobID is unavailable without
> "hacks" at this moment.
>
>
> > And is there any follow-up JIRA on the connector side to adopt the new interface?
>
> I've created a follow-up umbrella ticket
> https://issues.apache.org/jira/browse/FLINK-39827 . I have plans to
> adopt the proposed API in multiple places/different repos. Happy to
> elaborate if it makes sense at this moment.
>
> On Tue, 2 Jun 2026 at 18:00, Rui Fan <19...@gmail.com> wrote:
> >
> > Hey Aleksandr,
> >
> > Thanks for driving this — exposing JobInfo to the source context makes
> > sense, and it's already there on the sink side[1].
> >
> > A few questions:
> > 1. On the "no stability contract" argument for RichSourceReaderContext:
> >
> > Per FLIP-321[2], an @Experimental API should be promoted after two
> > releases. RichSourceReaderContext is well past that, so its @Experimental
> > tag
> > is a stalled promotion, not a genuine lack of stability contract. The real
> > gaps are
> > that it only exists on the reader side, and that access goes through the
> > whole
> > RuntimeContext. Worth reframing the motivation around those instead.
> >
> > 2. On the use cases:
> >
> > Is any of them a real connector today that's actually working around this?
> > And is there any follow-up JIRA on the connector side to adopt the new
> > interface?
> >
> > Best,
> > Rui
> >
> > [1]
> > https://github.com/apache/flink/blob/a5a7925415bdf6f257b100cc27e9aeb19d096d04/flink-core/src/main/java/org/apache/flink/api/connector/sink2/InitContext.java#L52
> >
> > [2]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-321:+Introduce+an+API+deprecation+process
> >
> >
> > On Tue, Jun 2, 2026 at 4:55 PM Yuepeng Pan <hi...@gmail.com> wrote:
> >
> > > Hi Aleksandr Savonin,
> > >
> > > +1. That sounds reasonable to me.
> > >
> > >
> > > Best,
> > > Yuepeng Pan
> > >
> > > Aleksandr Savonin <sa...@gmail.com> 于2026年6月2日周二 22:21写道：
> > >
> > > > Hi everyone,
> > > > gently pinging this thread to see if anyone has thoughts or concerns
> > > > regarding this proposal.
> > > >
> > > > On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <sa...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I’d like to start a discussion about FLIP-583: Expose JobInfo on
> > > > > Source contexts [1].
> > > > > Source connectors often need job metadata (job ID, job name) for
> > > > > authentication, distributed tracing, metrics labeling, and other
> > > > > usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
> > > > >
> > > > > The current workarounds on the source side (e.g. parsing the job ID
> > > > > from metric group variables) are brittle/fragile and undiscoverable.
> > > > > The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
> > > > > `SourceReaderContext` and `SplitEnumeratorContext`, following the same
> > > > > pattern as already established in these interfaces. The runtime
> > > > > implementations delegate to `StreamingRuntimeContext` (reader side)
> > > > > and `OperatorCoordinator.Context` (enumerator side).
> > > > >
> > > > > Looking forward to your feedback.
> > > > >
> > > > > [1]
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
> > > > >
> > > > > --
> > > > > Kind regards,
> > > > > Aleksandr
> > > >
> > > >
> > > >
> > > > --
> > > > Kind regards,
> > > > Aleksandr
> > > >
> > >
>
>
>
> --
> Kind regards,
> Aleksandr
--
Kind regards,
Aleksandr
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Aleksandr Savonin <sa...@gmail.com>.
Hi.
Thanks Yuepeng Pan and Rui for the feedback and participation!
> RichSourceReaderContext is well past that, so its @Experimental tag is a stalled promotion, not a genuine lack of stability contract.
Agree, I updated my FLIP (Motivation section) accordingly.
> Is any of them a real connector today that's actually working around this?
I've checked some of them, but could not find a connector working
around this today, likely because the JobID is unavailable without
"hacks" at this moment.
> And is there any follow-up JIRA on the connector side to adopt the new interface?
I've created a follow-up umbrella ticket
https://issues.apache.org/jira/browse/FLINK-39827 . I have plans to
adopt the proposed API in multiple places/different repos. Happy to
elaborate if it makes sense at this moment.
On Tue, 2 Jun 2026 at 18:00, Rui Fan <19...@gmail.com> wrote:
>
> Hey Aleksandr,
>
> Thanks for driving this — exposing JobInfo to the source context makes
> sense, and it's already there on the sink side[1].
>
> A few questions:
> 1. On the "no stability contract" argument for RichSourceReaderContext:
>
> Per FLIP-321[2], an @Experimental API should be promoted after two
> releases. RichSourceReaderContext is well past that, so its @Experimental
> tag
> is a stalled promotion, not a genuine lack of stability contract. The real
> gaps are
> that it only exists on the reader side, and that access goes through the
> whole
> RuntimeContext. Worth reframing the motivation around those instead.
>
> 2. On the use cases:
>
> Is any of them a real connector today that's actually working around this?
> And is there any follow-up JIRA on the connector side to adopt the new
> interface?
>
> Best,
> Rui
>
> [1]
> https://github.com/apache/flink/blob/a5a7925415bdf6f257b100cc27e9aeb19d096d04/flink-core/src/main/java/org/apache/flink/api/connector/sink2/InitContext.java#L52
>
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-321:+Introduce+an+API+deprecation+process
>
>
> On Tue, Jun 2, 2026 at 4:55 PM Yuepeng Pan <hi...@gmail.com> wrote:
>
> > Hi Aleksandr Savonin,
> >
> > +1. That sounds reasonable to me.
> >
> >
> > Best,
> > Yuepeng Pan
> >
> > Aleksandr Savonin <sa...@gmail.com> 于2026年6月2日周二 22:21写道：
> >
> > > Hi everyone,
> > > gently pinging this thread to see if anyone has thoughts or concerns
> > > regarding this proposal.
> > >
> > > On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <sa...@gmail.com>
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I’d like to start a discussion about FLIP-583: Expose JobInfo on
> > > > Source contexts [1].
> > > > Source connectors often need job metadata (job ID, job name) for
> > > > authentication, distributed tracing, metrics labeling, and other
> > > > usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
> > > >
> > > > The current workarounds on the source side (e.g. parsing the job ID
> > > > from metric group variables) are brittle/fragile and undiscoverable.
> > > > The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
> > > > `SourceReaderContext` and `SplitEnumeratorContext`, following the same
> > > > pattern as already established in these interfaces. The runtime
> > > > implementations delegate to `StreamingRuntimeContext` (reader side)
> > > > and `OperatorCoordinator.Context` (enumerator side).
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
> > > >
> > > > --
> > > > Kind regards,
> > > > Aleksandr
> > >
> > >
> > >
> > > --
> > > Kind regards,
> > > Aleksandr
> > >
> >
--
Kind regards,
Aleksandr
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Rui Fan <19...@gmail.com>.
Hey Aleksandr,
Thanks for driving this — exposing JobInfo to the source context makes
sense, and it's already there on the sink side[1].
A few questions:
1. On the "no stability contract" argument for RichSourceReaderContext:
Per FLIP-321[2], an @Experimental API should be promoted after two
releases. RichSourceReaderContext is well past that, so its @Experimental
tag
is a stalled promotion, not a genuine lack of stability contract. The real
gaps are
that it only exists on the reader side, and that access goes through the
whole
RuntimeContext. Worth reframing the motivation around those instead.
2. On the use cases:
Is any of them a real connector today that's actually working around this?
And is there any follow-up JIRA on the connector side to adopt the new
interface?
Best,
Rui
[1]
https://github.com/apache/flink/blob/a5a7925415bdf6f257b100cc27e9aeb19d096d04/flink-core/src/main/java/org/apache/flink/api/connector/sink2/InitContext.java#L52
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321:+Introduce+an+API+deprecation+process
On Tue, Jun 2, 2026 at 4:55 PM Yuepeng Pan <hi...@gmail.com> wrote:
> Hi Aleksandr Savonin,
>
> +1. That sounds reasonable to me.
>
>
> Best,
> Yuepeng Pan
>
> Aleksandr Savonin <sa...@gmail.com> 于2026年6月2日周二 22:21写道：
>
> > Hi everyone,
> > gently pinging this thread to see if anyone has thoughts or concerns
> > regarding this proposal.
> >
> > On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <sa...@gmail.com>
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > I’d like to start a discussion about FLIP-583: Expose JobInfo on
> > > Source contexts [1].
> > > Source connectors often need job metadata (job ID, job name) for
> > > authentication, distributed tracing, metrics labeling, and other
> > > usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
> > >
> > > The current workarounds on the source side (e.g. parsing the job ID
> > > from metric group variables) are brittle/fragile and undiscoverable.
> > > The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
> > > `SourceReaderContext` and `SplitEnumeratorContext`, following the same
> > > pattern as already established in these interfaces. The runtime
> > > implementations delegate to `StreamingRuntimeContext` (reader side)
> > > and `OperatorCoordinator.Context` (enumerator side).
> > >
> > > Looking forward to your feedback.
> > >
> > > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
> > >
> > > --
> > > Kind regards,
> > > Aleksandr
> >
> >
> >
> > --
> > Kind regards,
> > Aleksandr
> >
>
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Yuepeng Pan <hi...@gmail.com>.
Hi Aleksandr Savonin,
+1. That sounds reasonable to me.
Best,
Yuepeng Pan
Aleksandr Savonin <sa...@gmail.com> 于2026年6月2日周二 22:21写道：
> Hi everyone,
> gently pinging this thread to see if anyone has thoughts or concerns
> regarding this proposal.
>
> On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <sa...@gmail.com>
> wrote:
> >
> > Hi everyone,
> >
> > I’d like to start a discussion about FLIP-583: Expose JobInfo on
> > Source contexts [1].
> > Source connectors often need job metadata (job ID, job name) for
> > authentication, distributed tracing, metrics labeling, and other
> > usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
> >
> > The current workarounds on the source side (e.g. parsing the job ID
> > from metric group variables) are brittle/fragile and undiscoverable.
> > The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
> > `SourceReaderContext` and `SplitEnumeratorContext`, following the same
> > pattern as already established in these interfaces. The runtime
> > implementations delegate to `StreamingRuntimeContext` (reader side)
> > and `OperatorCoordinator.Context` (enumerator side).
> >
> > Looking forward to your feedback.
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
> >
> > --
> > Kind regards,
> > Aleksandr
>
>
>
> --
> Kind regards,
> Aleksandr
>
Re: [DISCUSS] FLIP-583: Expose JobInfo on Source contexts
Posted by Aleksandr Savonin <sa...@gmail.com>.
Hi everyone,
gently pinging this thread to see if anyone has thoughts or concerns
regarding this proposal.
On Wed, 27 May 2026 at 15:57, Aleksandr Savonin <sa...@gmail.com> wrote:
>
> Hi everyone,
>
> I’d like to start a discussion about FLIP-583: Expose JobInfo on
> Source contexts [1].
> Source connectors often need job metadata (job ID, job name) for
> authentication, distributed tracing, metrics labeling, and other
> usages. SinkV2 already exposes this via `InitContext.getJobInfo()`.
>
> The current workarounds on the source side (e.g. parsing the job ID
> from metric group variables) are brittle/fragile and undiscoverable.
> The FLIP adds a `@PublicEvolving` default method `getJobInfo()` to
> `SourceReaderContext` and `SplitEnumeratorContext`, following the same
> pattern as already established in these interfaces. The runtime
> implementations delegate to `StreamingRuntimeContext` (reader side)
> and `OperatorCoordinator.Context` (enumerator side).
>
> Looking forward to your feedback.
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-583%3A+Expose+JobInfo+on+Source+contexts
>
> --
> Kind regards,
> Aleksandr
--
Kind regards,
Aleksandr