You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gaborgsomogyi (via GitHub)" <gi...@apache.org> on 2026/08/28 15:35:41 UTC
[PR] [FLINK-40506][s3] Shade all runtime deps in flink-s3-fs-native to avoid silently dropping plugin dependencies [flink]
gaborgsomogyi opened a new pull request, #29042:
URL: https://github.com/apache/flink/pull/29042
## What is the purpose of the change
`flink-s3-fs-native`'s shade-plugin configuration uses an explicit `artifactSet` allowlist of dependency groups to bundle into the plugin's shaded jar, instead of including everything by default (as `flink-s3-fs-hadoop` does). This means any dependency that isn't in the allowlist is silently excluded from the packaged jar at build time, with no compile-time or test-time signal — the module still compiles and unit tests still pass, because the classes are present on the unshaded classpath.
This is not just a theoretical risk: `commons-codec`, a transitive runtime dependency of `org.apache.httpcomponents:httpclient` (itself a dependency of the bundled AWS SDK Apache HTTP client), is missing from the allowlist today and is therefore absent from the packaged plugin jar, even though it's present on the module's compile/runtime classpath. Any code path that ends up needing a `commons-codec` class at runtime (e.g. certain HTTP client authentication/encoding paths) would fail with a `NoClassDefFoundError` inside the plugin's isolated classloader, which cannot fall back to the Flink lib or user classpath to resolve it. This is hard to catch during review or testing, since the allowlist and the module's actual dependency tree live in different parts of the pom with nothing forcing them to stay in sync — every new (transitive) dependency needs a matching allowlist entry or it silently disappears from the shaded jar.
This pull request switches `flink-s3-fs-native`'s shade `artifactSet` to include all dependencies (`*:*`), matching `flink-s3-fs-hadoop`'s existing approach, so the shaded jar always reflects the module's actual dependency tree and future dependency changes can't silently regress plugin functionality.
## Brief change log
- Changed the `maven-shade-plugin` `artifactSet` in `flink-s3-fs-native/pom.xml` from an explicit list of included dependency groups to `*:*`, so all compile/runtime-scope dependencies are bundled into the shaded jar
## Verifying this change
This change is already covered by existing tests, such as the module's existing unit test suite.
- Manually verified by inspecting the built shaded jar before and after the change: `org/apache/commons/codec/**` (and other transitive dependency classes not covered by the previous allowlist) were entirely absent from the packaged jar before this change, despite being present on the module's compile/runtime classpath; after the change, all of the module's declared dependencies, including their transitive dependencies, are present in the shaded jar
- Ran the full `flink-s3-fs-native` unit test suite; all tests pass
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Claude code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40506][s3] Shade all runtime deps in flink-s3-fs-native to avoid silently dropping plugin dependencies [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29042:
URL: https://github.com/apache/flink/pull/29042#issuecomment-5454605425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "832b55d2aa04d18968ff4916e8e9635fbe260901",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "832b55d2aa04d18968ff4916e8e9635fbe260901",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 832b55d2aa04d18968ff4916e8e9635fbe260901 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40506][s3] Shade all runtime deps in flink-s3-fs-native to avoid silently dropping plugin dependencies [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #29042:
URL: https://github.com/apache/flink/pull/29042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40506][s3] Shade all runtime deps in flink-s3-fs-native to avoid silently dropping plugin dependencies [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #29042:
URL: https://github.com/apache/flink/pull/29042#issuecomment-5454573141
cc @Samrat002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org