You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "loustler (via GitHub)" <gi...@apache.org> on 2026/08/22 05:12:51 UTC

[PR] [Improve][Shade] Upgrade hadoop3-uber and hadoop-aws to 3.4.3, migrate S3A to AWS SDK v2, and relocate Jackson [seatunnel-shade]

loustler opened a new pull request, #20:
URL: https://github.com/apache/seatunnel-shade/pull/20

   @hawk9821 @davidzollo — this carries two changes that were made against `apache/seatunnel`'s in-repo `seatunnel-shade/` tree and now have to live here instead.
   
   ### Why it's here
   
   - apache/seatunnel#11673 relocated Jackson in `seatunnel-hadoop-aws` to match the uber jar. Merged there as `2586f37e96df`.
   - apache/seatunnel#11851 reverted it, because shade had moved to this repo. Its follow-up note asks for the fix to be resubmitted here, which is what commit 1 does.
   - apache/seatunnel#9993 then landed the actual cutover — `seatunnel-shade/` is gone from `apache/seatunnel` and consumers pull `seatunnel-shade-*` from Central.
   - apache/seatunnel#11648 is my open PR upgrading the shaded Hadoop to 3.4.3 and moving S3A to AWS SDK v2. Three of its 88 files were shade poms; those are commit 2 here. The other 85 (connectors, dist, e2e, docs) stay in `apache/seatunnel` and are blocked until this releases.
   
   ### Commit 1 — Jackson relocation in `seatunnel-shade-hadoop-aws`
   
   `hadoop-aws` carries no Jackson classes of its own. It references them through `hadoop-common`, which this jar does not bundle either — `seatunnel-shade-hadoop3-uber` supplies it, and that jar already relocates `com.fasterxml.jackson` to `${seatunnel.shade.package}.hadoop.com.fasterxml.jackson`. So `JsonSerialization#getMapper` is declared to return the relocated `ObjectMapper`.
   
   Left unrelocated here, `RoleModel`'s call site keeps the original descriptor `()Lcom/fasterxml/jackson/databind/ObjectMapper;` and fails at runtime with `NoSuchMethodError` on any `fs.s3a.assumed.role.*` path.
   
   The relocation uses the **uber jar's** shaded pattern, not this module's own prefix — the point is to agree with the class that supplies the method. Nothing is added to the jar; only references are rewritten.
   
   ### Commit 2 — Hadoop 3.4.3 and AWS SDK v2
   
   **Both modules move together.** `hadoop-aws` declares `hadoop-common` as `provided`, so at runtime it links against whatever `hadoop-common` the uber jar ships. `hadoop-aws` 3.1.4 against a 3.4.x uber jar fails with `NoClassDefFoundError org/apache/commons/lang/StringUtils` in `S3AFileSystem.initialize()` (hadoop-common dropped commons-lang 2.x after 3.1.4) and `NoSuchMethodError` on `SemaphoredDelegatingExecutor.<init>` in `S3AFileSystem.create()` (the guava-typed constructor was removed).
   
   **Version placement.** Both bumps are declared in the child modules' own `<properties>`, not in the parent, following the incremental-release rule in `README.md` — the parent is published at 3.0.0 and its properties are frozen. Happy to move them to the root pom if you'd rather do a full re-release.
   
   **AWS SDK bundle filtering.** `hadoop-aws` 3.4.x depends on `software.amazon.awssdk:bundle`, which is 686,119,507 bytes and carries all 410 AWS services. Resolving every `software/amazon/awssdk` reference in the 466 classes of `hadoop-aws` 3.4.3 — as type references and again as dotted strings, to catch reflective lookups — yields exactly three: `s3`, `sts`, `kms` (the last only `KmsClient`/`KmsClientBuilder`, from `EncryptionS3ClientFactory`). No `dynamodb`; S3Guard was removed in Hadoop 3.4.0.
   
   An allowlist filter keeps those three plus the 25 non-service packages, dropping 380,121 entries and 564,452,812 compressed bytes. The shaded jar comes out at **23,231,873 bytes instead of roughly 0.6 GB** — worth caring about for something ASF mirrors and archives indefinitely.
   
   Two things in that filter are deliberate and easy to "simplify" wrongly, so they're commented in the pom:
   
   1. It's an allowlist, not a denylist. maven-shade's `SimpleFilter` is `!(isIncluded && !isExcluded)`, so an exclude can never be re-included and "everything except these services" isn't expressible.
   2. The bundle is filtered rather than replaced by the individual `s3`/`sts`/`kms`/`apache-client` modules, because `hadoop-aws` is compiled against the bundle's *relocated* Apache HTTP classes — `ConfigureShadedAWSSocketFactory` references `software/amazon/awssdk/thirdparty/org/apache/http/...`, which standalone `apache-client` doesn't carry. `NetworkBinding` loads that class reflectively and swallows the failure, so the substitution wouldn't fail the build; it would quietly turn `fs.s3a.ssl.channel.mode` into a no-op.
   
   **SLF4J binding.** `hadoop-client` 3.3.3+ pulls `org.slf4j:slf4j-reload4j` through `hadoop-common` (HADOOP-18088), and Hadoop's own "no slf4j backends for downstream clients" exclusion list still names only `slf4j-log4j12`. This project manages no SLF4J binding, so the uber jar would package `org/slf4j/impl/**` — `apache/seatunnel`'s root pom used to keep that out, and that safety net doesn't exist here. SLF4J would bind to an unconfigured log4j 1.x whose root logger defaults to DEBUG, Parquet would install `RecordConsumerLoggingWrapper`, and every Parquet writer thread in the JVM would serialise on `Category.callAppenders`.
   
   Guarded twice: exclusions on the `hadoop-client` dependency, and a jar-level `org/slf4j/impl/**` filter in both modules. Only the binding, never the API — and the pattern is anchored at the jar root, so the SDK's own relocated copy under `software/amazon/awssdk/thirdparty/org/slf4j/**` is kept (inert for binding resolution, and the SDK needs it).
   
   ### Release note
   
   This produces two new coordinates, so there's no Nexus conflict with what's already published:
   
   - `seatunnel-shade-hadoop3-uber:3.4.3-3.0.0`
   - `seatunnel-shade-hadoop-aws:3.4.3-3.0.0`
   
   ```bash
   mvn -B -DskipTests -Drat.skip=true \
     -pl seatunnel-shade-hadoop3-uber,seatunnel-shade-hadoop-aws \
     clean deploy
   ```
   
   apache/seatunnel#11648 can't build until those are on Central — after that it's a two-property bump (`seatunnel.shade.hadoop.version`, `seatunnel.shade.hadoop-aws.version`) plus the non-shade files. If you can give me a rough sense of the release turnaround I'll plan that PR around it.
   
   ### How it was verified
   
   This repo has no CI, so everything below was run by hand on JDK 11:
   
   ```bash
   mvn -B -DskipTests -Drat.skip=true -Dgpg.skip=true \
     -pl seatunnel-shade-hadoop3-uber,seatunnel-shade-hadoop-aws clean install
   ```
   
   | Check | Result |
   |---|---|
   | Build | `BUILD SUCCESS`; `seatunnel-shade-hadoop3-uber-3.4.3-3.0.0.jar` (64,690,345 B), `seatunnel-shade-hadoop-aws-3.4.3-3.0.0.jar` (23,231,873 B) |
   | `maven-shade-plugin` 3.4.1 vs bcprov's Java 21 classes | fine — the existing `META-INF/versions/21/**` exclusion from #16 covers `bcprov-jdk18on`; no plugin bump needed |
   | SLF4J binding at jar root | 0 entries under `org/slf4j/impl/` in both jars; SLF4J API still present in the uber jar (38 entries) |
   | SDK's relocated slf4j | 16 entries kept, all under `software/amazon/awssdk/thirdparty/org/slf4j/impl/` |
   | AWS services in hadoop-aws jar | exactly `s3`, `sts`, `kms` |
   | Jackson descriptors agree | uber's `JsonSerialization#getMapper()` returns `org.apache.seatunnel.shade.hadoop.com.fasterxml.jackson.databind.ObjectMapper`; hadoop-aws's `RoleModel` call site is `…getMapper:()Lorg/apache/seatunnel/shade/hadoop/com/fasterxml/jackson/databind/ObjectMapper;` |
   | Unrelocated Jackson refs | 0 — all 13 occurrences across hadoop-aws's 466 classes are relocated; 0 Jackson classes bundled |
   
   ### Not included
   
   #11673 also added `tools/dependencies/check_shaded_jackson_refs.py` and a CI step that asserted the descriptor agreement above. There's no `.github/` in this repo, so I left it out rather than adding a script nothing runs. Glad to bring it over if you'd like to introduce CI here — it's the check I ran by hand for the last two table rows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org