You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by archit goyal <go...@gmail.com> on 2026/02/02 17:38:04 UTC
Re: [DISCUSS] FLIP-563: Support provided lib archives for YARN application mode
Hi all,
Gentle ping on FLIP-563 - would appreciate reviews/feedback from the community.
On Wed, Jan 21, 2026 at 3:29 PM archit goyal <go...@gmail.com> wrote:
>
> Hi All,
>
> I am opening this thread to discuss FLIP-563: Support for provided lib archives in YARN Application Mode.
>
> Today yarn.provided.lib.dirs assumes pre-uploaded directories of jars. In practice, many deployments package shared libs as .tgz/.tar.gz and/or hit classpath “argument too long” issues when many jars are enumerated.
>
> Proposal: introduce yarn.provided.lib.archives to point to pre-uploaded HDFS archives. Flink would localize them as public archive resources and add a single wildcard classpath entry per archive (e.g., <archiveDir>/*). Backward compatible and no change unless configured.
>
> Please feel free to reply to this email thread and share your opinions.
>
> Thanks,
> Archit Goyal
--
Yours Sincerely
Archit Goyal
Email: goyalarchit1@gmail.com
Mobile: +1-347-471-9251
Re: [DISCUSS] FLIP-563: Support provided lib archives for YARN application mode
Posted by archit goyal <go...@gmail.com>.
Hi Samrat,
Thanks for the review comments. Please find my response below:
1. *yarn.provided.lib.dirs is not merely an unordered collection of
jars...Can you elaborate on what archive replacement will look like? *
Agree that a flat wildcard cannot replicate flink-dist/plugin/resource-dir
handling. Rather than trying to extend archives to cover all of that, I am
scoping yarn.provided.lib.archives narrowly as a wildcard classpath
mechanism for jars only. Non-jar resources, flink-dist, and plugins remain
the responsibility of yarn.provided.lib.dirs. Archives are additive
alongside lib.dirs, not a replacement, so this is not a loss of capability.
2. *The JDK launcher specification explicitly states...How will this be
handled deterministically? *
Since flink-dist stays out of archives entirely, it keeps its current
explicit, ordering-guaranteed handling via lib.dirs, unaffected by wildcard
ordering. For jars inside an archive, ordering is genuinely unspecified per
the JDK wildcard spec, and there is no way to fix that client-side since
Flink never sees individual jar names before the NodeManager unpacks the
archive. I have documented this as a hard constraint: jars in a provided
archive must not conflict with each other or with lib.dirs jars (duplicate
classes, service providers, logging bindings) which is the same
responsibility users already have today avoiding collisions across multiple
lib.dirs entries.
3. *Every archive descriptor must reach both the AM and later
TaskManagers...If a local distribution is retained, how are duplicate Flink
jars prevented? *
flink-dist is unaffected by this proposal and it is never placed inside an
archive's wildcard entry. I have added a validation step where submission
fails fast if yarn.provided.lib.archives is set with no flink-dist source
available anywhere (no lib.dirs, no local distribution).
4. *The localised resource name is part of both the YARN resource map and
the classpath...Will there be any silent override?*
This is a real gap regardless of scoping as of
today, getAllFilesInProvidedLibDirs dedupes by filename across lib.dirs and
throws on collision, but there is no equivalent for multiple archives
resolving to the same localized resource name. I have added that as an
explicit validation failure in the FLIP.
5. *Thinking loud from the perspective of security, archives are
executable...Can you shed some light on YARN Private and Public only in the
flip?*
Archives use the same PUBLIC visibility model as lib.dirs today ( compared
to PRIVATE / APPLICATION for regular per-application resources, which are
not cached/reused across apps). World readability is documented but not
enforced by Flink, consistent with existing behavior. Unpacking is
delegated entirely to YARN NodeManager, not done by Flink. I have flagged
in the FLIP that deeper YARN-internals security review might be needed and
welcomed reviewers with that background to help review.
6. *Can you cover the test plan and alternatives in the flip with more
detail? *
Expanded the test plan to cover the new validation failures (missing dist
source, cross-archive name collision), classpath/wildcard registration
checks, and the jars-only exclusion for flink-dist/plugins/non-jar content,
plus an integration test submitting a real YARN job with an archive. Also,
added two Rejected Alternatives: client-side pre-extraction (defeats
NodeManager caching, can reintroduce the long-classpath problem), and
extending wildcard support directly to lib.dirs (doesn't address the
motivation of shipping a single pre-packaged archive).
FLIP is updated with all of the above. Happy to adjust further based on the
feedback.
Thanks,
Archit
On Wed, Aug 12, 2026 at 10:56 AM Samrat Deb <de...@gmail.com> wrote:
> Hi Archit,
>
> Thanks for proposing this. Overall +1. Proposal looks feasible. YARN
> supports PUBLIC ARCHIVE resources, Flink’s resource descriptor already
> preserves visibility and resource type when resources are propagated to
> TaskManagers, and the Java launcher expands wildcards in `CLASSPATH`.
>
> Below are a few queries on the proposal
>
> 1. `yarn.provided.lib.dirs` is not merely an unordered collection of jars.
> The current deployment path treats several resources specially:
> - `flink-dist` is identified and propagated separately to dynamically
> allocated TaskManagers.
> - Plugin jars are deliberately excluded from the system classpath and
> remain under the plugin classloader hierarchy.
> - Non-jar resource directories can be added to the classpath.
> - Local `lib/` and `plugins/` shipping is suppressed when a complete
> provided distribution is used.
> - User jars remain governed by `yarn.classpath.include-user-jar`.
> A flat `<archive>/*` entry does not preserve these semantics. It would load
> every top-level jar through the system classloader, exclude loose
> resources, and provide no plugin isolation.
> Can you elaborate on what archive replacement will look like?
>
> 2. The JDK launcher specification explicitly states that the order of jars
> produced by directory/* is unspecified. This matters because the current
> Flink startup scripts sort jars and place flink-dist separately, and the
> YARN implementation similarly sorts system classpath entries and appends
> flink-dist last. Replacing this with a wildcard can resolve duplicate
> classes, service providers, logging bindings, and resources that depend on
> the NodeManager, filesystem, or JDK implementation. Including flink-dist in
> that wildcard is especially concerning because its current ordering would
> be lost. How will this be handled deterministically?
>
> 3. Every archive descriptor must reach both the AM and later TaskManagers.
> The existing TaskManager launch path requires _FLINK_DIST_JAR and
> reconstructs other resources from _CLIENT_SHIP_FILES. For archive-only
> deployments, which resource satisfies _FLINK_DIST_JAR? If a local
> distribution is retained, how are duplicate Flink jars prevented?
>
> 4. The localised resource name is part of both the YARN resource map and
> the classpath. Two paths such as /v1/flink-libs.tgz and /v2/flink-libs.tgz
> have the same basename. Will there be any silent override?
>
> 5. Thinking loud from the perspective of security, archives are executable.
> Are there any known gotchas? Can you shed some light on YARN Private and
> Public only in the flip?
>
> 6. Can you cover the test plan and alternatives in the flip with more
> detail, if there are any?
>
> Bests,
> Samrat
>
> On Mon, Aug 10, 2026 at 11:10 PM archit goyal <go...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Gentle bump on the thread for FLIP-563 review to support
> > *yarn.provided.lib.archives* to point to pre-uploaded HDFS archives.
> >
> > On Tue, Aug 4, 2026 at 9:28 PM archit goyal <go...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives*
> to
> > > point to pre-uploaded HDFS archives.
> > >
> > > Thanks,
> > > Archit Goyal
> > >
> > > On Mon, Feb 2, 2026 at 9:38 AM archit goyal <go...@gmail.com>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the
> > >> community.
> > >>
> > >> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <go...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi All,
> > >> >
> > >> > I am opening this thread to discuss FLIP-563: Support for provided
> lib
> > >> archives in YARN Application Mode.
> > >> >
> > >> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of
> jars.
> > >> In practice, many deployments package shared libs as .tgz/.tar.gz
> and/or
> > >> hit classpath “argument too long” issues when many jars are
> enumerated.
> > >> >
> > >> > Proposal: introduce yarn.provided.lib.archives to point to
> > pre-uploaded
> > >> HDFS archives. Flink would localize them as public archive resources
> and
> > >> add a single wildcard classpath entry per archive (e.g.,
> > <archiveDir>/*).
> > >> Backward compatible and no change unless configured.
> > >> >
> > >> > Please feel free to reply to this email thread and share your
> > opinions.
> > >> >
> > >> > Thanks,
> > >> > Archit Goyal
> > >>
> > >>
> > >>
> > >> --
> > >> Yours Sincerely
> > >> Archit Goyal
> > >> Email: goyalarchit1@gmail.com
> > >> Mobile: +1-347-471-9251
> > >>
> > >
> > >
> > > --
> > > Yours Sincerely
> > > Archit Goyal
> > > Email: goyalarchit1@gmail.com
> > >
> >
> >
> > --
> > Yours Sincerely
> > Archit Goyal
> > Email: goyalarchit1@gmail.com
> >
>
--
Yours Sincerely
Archit Goyal
Email: goyalarchit1@gmail.com
Re: [DISCUSS] FLIP-563: Support provided lib archives for YARN application mode
Posted by Samrat Deb <de...@gmail.com>.
Hi Archit,
Thanks for proposing this. Overall +1. Proposal looks feasible. YARN
supports PUBLIC ARCHIVE resources, Flink’s resource descriptor already
preserves visibility and resource type when resources are propagated to
TaskManagers, and the Java launcher expands wildcards in `CLASSPATH`.
Below are a few queries on the proposal
1. `yarn.provided.lib.dirs` is not merely an unordered collection of jars.
The current deployment path treats several resources specially:
- `flink-dist` is identified and propagated separately to dynamically
allocated TaskManagers.
- Plugin jars are deliberately excluded from the system classpath and
remain under the plugin classloader hierarchy.
- Non-jar resource directories can be added to the classpath.
- Local `lib/` and `plugins/` shipping is suppressed when a complete
provided distribution is used.
- User jars remain governed by `yarn.classpath.include-user-jar`.
A flat `<archive>/*` entry does not preserve these semantics. It would load
every top-level jar through the system classloader, exclude loose
resources, and provide no plugin isolation.
Can you elaborate on what archive replacement will look like?
2. The JDK launcher specification explicitly states that the order of jars
produced by directory/* is unspecified. This matters because the current
Flink startup scripts sort jars and place flink-dist separately, and the
YARN implementation similarly sorts system classpath entries and appends
flink-dist last. Replacing this with a wildcard can resolve duplicate
classes, service providers, logging bindings, and resources that depend on
the NodeManager, filesystem, or JDK implementation. Including flink-dist in
that wildcard is especially concerning because its current ordering would
be lost. How will this be handled deterministically?
3. Every archive descriptor must reach both the AM and later TaskManagers.
The existing TaskManager launch path requires _FLINK_DIST_JAR and
reconstructs other resources from _CLIENT_SHIP_FILES. For archive-only
deployments, which resource satisfies _FLINK_DIST_JAR? If a local
distribution is retained, how are duplicate Flink jars prevented?
4. The localised resource name is part of both the YARN resource map and
the classpath. Two paths such as /v1/flink-libs.tgz and /v2/flink-libs.tgz
have the same basename. Will there be any silent override?
5. Thinking loud from the perspective of security, archives are executable.
Are there any known gotchas? Can you shed some light on YARN Private and
Public only in the flip?
6. Can you cover the test plan and alternatives in the flip with more
detail, if there are any?
Bests,
Samrat
On Mon, Aug 10, 2026 at 11:10 PM archit goyal <go...@gmail.com>
wrote:
> Hi All,
>
> Gentle bump on the thread for FLIP-563 review to support
> *yarn.provided.lib.archives* to point to pre-uploaded HDFS archives.
>
> On Tue, Aug 4, 2026 at 9:28 PM archit goyal <go...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives* to
> > point to pre-uploaded HDFS archives.
> >
> > Thanks,
> > Archit Goyal
> >
> > On Mon, Feb 2, 2026 at 9:38 AM archit goyal <go...@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the
> >> community.
> >>
> >> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <go...@gmail.com>
> >> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > I am opening this thread to discuss FLIP-563: Support for provided lib
> >> archives in YARN Application Mode.
> >> >
> >> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of jars.
> >> In practice, many deployments package shared libs as .tgz/.tar.gz and/or
> >> hit classpath “argument too long” issues when many jars are enumerated.
> >> >
> >> > Proposal: introduce yarn.provided.lib.archives to point to
> pre-uploaded
> >> HDFS archives. Flink would localize them as public archive resources and
> >> add a single wildcard classpath entry per archive (e.g.,
> <archiveDir>/*).
> >> Backward compatible and no change unless configured.
> >> >
> >> > Please feel free to reply to this email thread and share your
> opinions.
> >> >
> >> > Thanks,
> >> > Archit Goyal
> >>
> >>
> >>
> >> --
> >> Yours Sincerely
> >> Archit Goyal
> >> Email: goyalarchit1@gmail.com
> >> Mobile: +1-347-471-9251
> >>
> >
> >
> > --
> > Yours Sincerely
> > Archit Goyal
> > Email: goyalarchit1@gmail.com
> >
>
>
> --
> Yours Sincerely
> Archit Goyal
> Email: goyalarchit1@gmail.com
>
[DISCUSS] FLIP-563: Support provided lib archives for YARN application mode
Posted by archit goyal <go...@gmail.com>.
Hi All,
Gentle bump on the thread for FLIP-563 review to support
*yarn.provided.lib.archives* to point to pre-uploaded HDFS archives.
On Tue, Aug 4, 2026 at 9:28 PM archit goyal <go...@gmail.com> wrote:
> Hi All,
>
> Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives* to
> point to pre-uploaded HDFS archives.
>
> Thanks,
> Archit Goyal
>
> On Mon, Feb 2, 2026 at 9:38 AM archit goyal <go...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the
>> community.
>>
>> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <go...@gmail.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > I am opening this thread to discuss FLIP-563: Support for provided lib
>> archives in YARN Application Mode.
>> >
>> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of jars.
>> In practice, many deployments package shared libs as .tgz/.tar.gz and/or
>> hit classpath “argument too long” issues when many jars are enumerated.
>> >
>> > Proposal: introduce yarn.provided.lib.archives to point to pre-uploaded
>> HDFS archives. Flink would localize them as public archive resources and
>> add a single wildcard classpath entry per archive (e.g., <archiveDir>/*).
>> Backward compatible and no change unless configured.
>> >
>> > Please feel free to reply to this email thread and share your opinions.
>> >
>> > Thanks,
>> > Archit Goyal
>>
>>
>>
>> --
>> Yours Sincerely
>> Archit Goyal
>> Email: goyalarchit1@gmail.com
>> Mobile: +1-347-471-9251
>>
>
>
> --
> Yours Sincerely
> Archit Goyal
> Email: goyalarchit1@gmail.com
>
--
Yours Sincerely
Archit Goyal
Email: goyalarchit1@gmail.com
Re: [DISCUSS] FLIP-563: Support provided lib archives for YARN application mode
Posted by archit goyal <go...@gmail.com>.
Hi All,
Gentle bump on FLIP-563 review to support *yarn.provided.lib.archives* to
point to pre-uploaded HDFS archives.
Thanks,
Archit Goyal
On Mon, Feb 2, 2026 at 9:38 AM archit goyal <go...@gmail.com> wrote:
> Hi all,
>
> Gentle ping on FLIP-563 - would appreciate reviews/feedback from the
> community.
>
> On Wed, Jan 21, 2026 at 3:29 PM archit goyal <go...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > I am opening this thread to discuss FLIP-563: Support for provided lib
> archives in YARN Application Mode.
> >
> > Today yarn.provided.lib.dirs assumes pre-uploaded directories of jars.
> In practice, many deployments package shared libs as .tgz/.tar.gz and/or
> hit classpath “argument too long” issues when many jars are enumerated.
> >
> > Proposal: introduce yarn.provided.lib.archives to point to pre-uploaded
> HDFS archives. Flink would localize them as public archive resources and
> add a single wildcard classpath entry per archive (e.g., <archiveDir>/*).
> Backward compatible and no change unless configured.
> >
> > Please feel free to reply to this email thread and share your opinions.
> >
> > Thanks,
> > Archit Goyal
>
>
>
> --
> Yours Sincerely
> Archit Goyal
> Email: goyalarchit1@gmail.com
> Mobile: +1-347-471-9251
>
--
Yours Sincerely
Archit Goyal
Email: goyalarchit1@gmail.com