You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "fatmanverse (via GitHub)" <gi...@apache.org> on 2026/07/15 07:17:14 UTC

[PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

fatmanverse opened a new pull request, #11436:
URL: https://github.com/apache/seatunnel/pull/11436

   ## Purpose of this pull request
   
   Closes #9233
   
   The S3File connector previously hard-coded the `fs.s3a.aws.credentials.provider` option as an enum (`S3aAwsCredentialsProvider`) that accepted **only** `SimpleAWSCredentialsProvider` and `InstanceProfileCredentialsProvider`. Any other value threw at config-parse time, and the value could not even be smuggled in via `hadoop_s3_properties` because the connector force-overwrote the key. This blocked users who need container/role-based providers — e.g. running SeaTunnel in K8s and authenticating to S3 via `WebIdentityTokenFileCredentialsProvider` (webIdentityTokenFile + role), which is exactly the scenario reported in #9233.
   
   This PR replaces the enum with a free-form string option that passes any fully-qualified S3A credentials provider class through to Hadoop, while keeping the two well-known values and the existing default.
   
   ## Changes
   
   - **`S3FileBaseOptions`**: `S3A_AWS_CREDENTIALS_PROVIDER` changed from `enumType(S3aAwsCredentialsProvider.class)` to `stringType()`. Default is unchanged (`com.amazonaws.auth.InstanceProfileCredentialsProvider`). The `S3aAwsCredentialsProvider` enum is removed; the two well-known class names are kept as `public static final String` constants (`SIMPLE_AWS_CREDENTIALS_PROVIDER`, `INSTANCE_PROFILE_CREDENTIALS_PROVIDER`) so factories and downstream modules keep compiling.
   - **`S3HadoopConf`**: passes the raw provider string through to `fs.s3a.aws.credentials.provider`, and eagerly verifies the class is on the classpath (`Class.forName`, init=false). The check tries the thread context classloader first, then falls back to the connector's own classloader, so a provider bundled with the connector (including the default) is never rejected by mistake under the Zeta plugin classloader. A missing class produces an actionable error pointing at the config key and `${SEATUNNEL_HOME}/lib`.
   - **Factories** (`S3FileSourceFactory`, `S3FileSinkFactory`, `S3RedshiftSinkFactory`): reference `SIMPLE_AWS_CREDENTIALS_PROVIDER` in the `.conditional(...)` rule that requires `access_key`/`secret_key`, instead of the removed enum value. Behavior is unchanged.
   - **Docs** (`docs/{en,zh}/connectors/{source,sink}/S3File.md`): the `fs.s3a.aws.credentials.provider` row now documents that any classpath-present S3A provider is accepted, with `ContainerCredentialsProvider` as a container example.
   - **Tests**: `S3HadoopConfTest` adds default / custom-passthrough / unknown-class-fails cases; `S3FileFactoryTest` asserts the conditional access_key/secret_key requirement still fires for both source and sink; `S3StorageTest` (hive) updated to the new constant.
   
   ## Backward compatibility
   
   Fully backward compatible. The old enum's `toString()` returned the fully-qualified class name, so existing configs already used exactly the strings the new `stringType()` accepts. The default value and the `access_key`/`secret_key` conditional requirement are unchanged.
   
   ## Does this PR introduce any user-facing change?
   
   Yes — `fs.s3a.aws.credentials.provider` now accepts any S3A credentials provider class available on the classpath (previously only two values). This is additive; existing configurations continue to work unchanged. Docs (en/zh) are updated.
   
   ## How was this patch tested?
   
   - `./mvnw spotless:apply` (clean)
   - Unit tests for `connector-file-s3` (`S3HadoopConfTest`, `S3FileFactoryTest`) and hive `S3StorageTest` — all passing.
   
   ## Check list
   
   - [x] Code changed are covered with tests, or it does not need tests for reason
   - [x] If any new Jar binary package adding in your PR, please add License Notice — N/A
   - [x] If necessary, please update the documentation to describe the new feature — updated en/zh
   - [ ] If you are contributing the connector code, please check that the following files are updated: N/A (no new connector)
   - [x] Update the `release-note`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436#issuecomment-5001891879

   Thanks for the thorough re-review — the runtime call-graph writeup made each point easy to reason about. Pushed a follow-up commit (`ef5ce98`). Here's how each item was addressed, plus one regression your Issue 3 analysis helped me catch.
   
   **Issue 1 (blocking) — binary compatibility.** Good catch on the type removal. To be precise about the break: the `S3A_AWS_CREDENTIALS_PROVIDER` field descriptor is actually unchanged (both `Option<Enum>` and `Option<String>` erase to the same descriptor), so field references stay binary-compatible; the only real removal was the `S3aAwsCredentialsProvider` enum *type* itself. I've restored that enum as `@Deprecated` (its constants now reference the new `String` constants, single source of truth), which removes the `NoClassDefFoundError` risk for any downstream code compiled against it. The option stays `stringType()` since that's the core of the feature. The source-level generic change is documented in `docs/{en,zh}/introduction/concepts/incompatible-changes.md` with migration guidance.
   
   **Issue 3 — check runs on the conf-building node, not workers. This turned out to be a real bug, not just a robustness concern.** Your analysis was exactly right, and it was already breaking CI: the original eager `Class.forName` check *hard-failed*, and because the default provider (`InstanceProfileCredentialsProvider`) lives in `seatunnel-hadoop-aws` at `provided` scope, it isn't on the classpath of the module that builds the conf in `connector-hive`'s unit tests — so `S3StorageTest` was failing on the *default* config (this is the `unit-test` job that was red). I've changed the check to **best-effort**: a class that cannot be resolved on the conf-building node is now only logged as a `WARN` (with the "must be present on every node running S3A" hint) and passed through, never rejected. `S3StorageTest` is green again.
   
   **Issue 2 — interface check.** Kept a *fast-fail* only for the case that can never work on any node: a class that *does* resolve but does not implement `com.amazonaws.auth.AWSCredentialsProvider` is rejected with an actionable error. The check also splits Hadoop-style comma-separated provider chains and validates each entry. The interface assertion is skipped when the AWS SDK interface itself doesn't resolve from the same classloader (plugin-classloader isolation), to avoid falsely rejecting a valid provider. On the **allowlist**: I'd prefer to leave it out of this PR. It's a new operator-facing config with its own compat surface, and the same class-name-loading vector already exists for any `*.class` key passed through `hadoop_s3_properties` — singling out this one option would be inconsistent. Happy to file a separate issue if a general "class-name option allowlist" is worth pursuing cluster-wide.
   
   **Issue 4 — docs.** Updated all four rows (en/zh × source/sink): the class must implement the AWS credentials-provider contract and expose a no-arg or `(URI, Configuration)` constructor; comma-separated chains are accepted; the jar must be on **every** node; and a short security note that this option lets job authors load classes by name on shared/multi-tenant clusters.
   
   On the runtime `ClassNotFoundException` wrapping and a dedicated Zeta e2e (Issue 3 follow-ups): I've left those out on purpose. The runtime path goes through `HadoopFileSystemProxy`, which is shared by all file connectors, so wrapping it there is a cross-connector change beyond this PR's scope; and a full Zeta e2e feels heavy for a non-blocking item. Both make sense as separate follow-ups if you'd like.
   
   Verified locally: `spotless:apply` clean, `connector-file-s3` unit tests pass, and `connector-hive` `S3StorageTest` passes again. Thanks again for the careful review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436#issuecomment-5011920947

   @SEZ9 I reworked the review fixes in `fcfcf014634d`; this comment supersedes my previous response.
   
   **Issues 1 and 4 - public API compatibility**
   
   The fix now preserves the original public API exactly, rather than only restoring the enum type:
   
   - `S3A_AWS_CREDENTIALS_PROVIDER` remains `Option<S3aAwsCredentialsProvider>` with the same key and default.
   - The deprecated `S3aAwsCredentialsProvider` enum remains available.
   - A separate `Option<String> S3A_AWS_CREDENTIALS_PROVIDER_CLASS` uses the same stable config key for the new free-form path.
   - All in-repo production consumers (S3 source/sink factories, Hive, Redshift, and `S3HadoopConf`) now use the new string option. The legacy field is only a compatibility surface and no longer owns runtime behavior.
   - The EN/ZH incompatible-change entry was removed because downstream code no longer needs recompilation or migration.
   
   I also added a compile-time regression assertion by assigning `ReadonlyConfig.get(S3A_AWS_CREDENTIALS_PROVIDER)` directly to `S3aAwsCredentialsProvider`. `javap` confirms that both public generic signatures are present.
   
   **Issue 2 - provider validation and classloader isolation**
   
   Each candidate classloader must now resolve both the configured class and `AWSCredentialsProvider` before assignability is checked. If an isolated TCCL resolves only one side, validation continues with the connector classloader instead of returning early. Resolvable non-provider and abstract classes fail fast; locally unresolvable classes remain a warning because the provider is instantiated on worker JVMs.
   
   **Issue 3 - Hadoop chain semantics**
   
   The validator now calls Hadoop 3.1.4 `StringUtils.getTrimmedStrings` directly. This matches Hadoop for comma/newline separators, whitespace, and a trailing comma, while still rejecting an interior empty entry such as `A,,B`. Tests cover whitespace, newline validation, trailing comma parity with `Configuration.getClasses`, interior empty entries, and the isolated-TCCL bypass.
   
   **Docs**
   
   All four EN/ZH source/sink rows now document the four creation mechanisms supported by Hadoop 3.1.4: `(URI, Configuration)`, `(Configuration)`, static no-arg `getInstance()`, and a no-arg constructor. They also retain the interface, provider-chain, every-node classpath, and shared-cluster security notes.
   
   **Local verification**
   
   - `S3HadoopConfTest,S3FileFactoryTest`: 17 tests passed
   - `S3StorageTest`: 2 tests passed
   - `connector-s3-redshift`: package passed
   - `./mvnw spotless:apply`: passed
   - `./mvnw -q -DskipTests verify`: passed across the full reactor
   
   I also attempted the root `./mvnw test`; it stops before the affected modules in `seatunnel-config-shade` with 7 unrelated `NoClassDefFoundError` failures for shaded Typesafe Config classes. The PR CI run for this head is now queued.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436#issuecomment-5010822442

   Thanks again for the careful re-review — really appreciate the runtime-chain writeup, it made each point easy to follow. Pushed a follow-up (`f563743`). Here's where each item stands:
   
   **Issue 2 (interface check) — addressed.** `S3HadoopConf` now asserts the resolved class `isAssignableFrom(com.amazonaws.auth.AWSCredentialsProvider)` and rejects abstract classes before writing the property. I cross-checked this against `hadoop-aws 3.1.4` (the version this repo pins, AWS SDK v1): these are the same two checks `S3AUtils.createAWSCredentialProvider` performs before instantiation (`NOT_AWS_PROVIDER` / `ABSTRACT_PROVIDER`), so the eager check should never reject anything Hadoop itself would accept — it just surfaces the failure at config-parse time with a clearer message. To stay safe under plugin-classloader isolation, the assertion is skipped when the AWS interface isn't visible from the same classloader that resolved the provider, so a valid provider is never falsely rejected.
   
   **Issue 3 (chain splitting) — addressed.** Splitting now uses `split(",", -1)` + `trim()` to mirror Hadoop's `getTrimmedStrings` semantics: `"A, B"` is accepted, while `"A,,B"` and a trailing `"A,"` are rejected as empty entries. The class-not-found path is a WARN rather than a hard failure, and it now explicitly says the jar must be present on **every** node. I added `S3HadoopConfTest` cases for `"A, B"`, `"A,,B"`, `"A,"`, chain pass-through, a non-provider entry within a chain, and an abstract class.
   
   **Issue 1 & 4 (enum removal / binary compatibility) — just want to make sure we're looking at the same state.** The `S3aAwsCredentialsProvider` enum is still present — it was restored as `@Deprecated` in `ef5ce98`, with its constants delegating to the new `String` constants as a single source of truth, so the `NoClassDefFoundError` risk for downstream code should be covered. The remaining change is the option's generic type (`Option<S3aAwsCredentialsProvider>` → `Option<String>`), which unfortunately can't coexist as two same-named fields; that's documented as a code-level (not just config-level) break in `incompatible-changes.md` (en + zh) with migration notes, and I confirmed no other module — including the Zeta checkpoint-storage S3 path — still references the removed generic type. If there's a specific angle on the binary-compatibility concern I've missed, I'd be glad to dig into it further.
   
   On the CI failures: I looked into these and they appear unrelated to the S3 change. `unit-test` fails in `seatunnel-engine-server`'s `MetricsApiTest` (an environment-dependent test that binds port 8080 / a Hazelcast HTTP endpoint), and `integration-test-part-5` fails in `FtpFileIT` (a Testcontainers FTP e2e that gets pulled in only because this PR touches a shared file-connector base module). Neither exercises the S3 credentials path, and this branch doesn't modify those modules. Happy to re-trigger a run.
   
   Verified locally: `spotless:apply` clean, and the `connector-file-s3` unit tests (including the new chain/interface/abstract cases) all pass. Thanks again for the guidance on this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436#issuecomment-5029663474

   @SEZ9 Merged the latest `dev` to clear the merge conflict (merge commit `3f3f86c1f`).
   
   The only conflict was in `connector-hive`'s `S3StorageTest.java`: `dev` (#11459) had refactored that test to extract a shared `TEST_S3_CREDENTIALS_PROVIDER` constant, while this PR had renamed the option to `S3A_AWS_CREDENTIALS_PROVIDER_CLASS` and flattened the enum into class-name constants — both touched the same lines.
   
   Resolution keeps `dev`'s shared-constant structure but points it at this PR's new API:
   
   ```java
   private static final String TEST_S3_CREDENTIALS_PROVIDER =
           S3FileBaseOptions.SIMPLE_AWS_CREDENTIALS_PROVIDER;
   // both configs now use S3A_AWS_CREDENTIALS_PROVIDER_CLASS.key()
   ```
   
   The two `S3File.md` doc conflicts auto-merged cleanly. Verified locally: `spotless:check` clean and `S3StorageTest` green (2/2). The PR is now mergeable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Support arbitrary S3A credentials provider for S3File [seatunnel]

Posted by "fatmanverse (via GitHub)" <gi...@apache.org>.
fatmanverse commented on PR #11436:
URL: https://github.com/apache/seatunnel/pull/11436#issuecomment-4987256084

   Thanks for the review, and for tracing the S3 call chain!
   
   I compared this branch against `origin/dev` (locally and via the GitHub API), and the diff shows 12 files, all S3-scoped — the four `S3File.md` docs, `S3FileBaseOptions`, `S3HadoopConf`, the two S3 factories, `S3RedshiftSinkFactory`, and the S3/hive tests that referenced the removed enum. I couldn't find `PostgresCatalog.java`, `JdbcPostgresIT.java`, or `JdbcOpenGaussIT.java` in it.
   
   I think the branch was just behind `dev` — the dropped-column `PostgresCatalog` fix (`d6df7f2`) landed there recently, so it likely showed up when diffing against an older base. I'll rebase onto the latest `dev` and force-push so the diff is clean; that should make the scope unambiguous. Thanks for your patience!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org