You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "hansva (via GitHub)" <gi...@apache.org> on 2026/10/17 20:19:41 UTC
[PR] Dependency upgrades and minor fixes, fixes #8966 (hop)
hansva opened a new pull request, #8997:
URL: https://github.com/apache/hop/pull/8997
## Dependency upgrades and lib/core cleanup
Upgrades third-party dependencies to their latest patch/minor releases, and removes duplicate and conflicting jars from the assemblies.
### Upgrades
- Patch/minor upgrades across `lib/pom.xml`, `lib-jdbc` and the plugin poms, among them the AWS SDK (2.20 → 2.56, now one version through the AWS BOM), Netty, OpenTelemetry, Jetty, Jersey, SWT/RAP, the JDBC drivers, Parquet, Tika 3.3.3 and OpenLineage 1.54.
- Driver download defaults in the database meta classes, the bundled-library versions in `LICENSE`, and the Snowflake docs page now match the bundled drivers.
- Spark, and the libraries Beam/Spark depend on through `lib/core` (Jackson, protobuf/gRPC, Hadoop, Kafka), are unchanged. ZooKeeper is the exception: it moves to 3.9.6, replacing the end-of-life 3.6.3 that Beam's Spark runner brought.
### One version per library in lib/core
`lib/core` is a single classloader, so two versions of a jar there means whichever is listed first wins.
- `lib/pom.xml` now manages the libraries that used to land in `lib/core` in several versions: google-auth, api-common, proto-google-common-protos, animal-sniffer, failureaccess, httpcore5, jaxb-api, jcl-over-slf4j, snappy-java, Parquet, Log4j (through `log4j-bom`) and more.
- Plugins no longer ship jars that `lib/core` already has. They declare them `provided` (JMS, Kafka, Lakehouse, OpenLineage, LanguageModelChat, Cassandra, Stanford NLP and others), or, for Spark, exclude them in the assembly. Spark keeps its own Spark 4 / Scala 2.13 stack.
- Native Spark uses Log4j and lz4 from `lib/core`. Spark's old `org.lz4:lz4-java` is replaced by the maintained `at.yawk.lz4` fork.
- Test jars no longer reach `lib/core`: JUnit 4.10 through `json-simple`, `reload4j` through Hadoop. Hamcrest stays: Beam's `PipelineOptionsFactory` needs it at startup.
- Library versions are aligned across plugins: jgit, antlr-runtime, libphonenumber, commons-pool2, jts-core.
### Fixes found by the integration tests
- **Beam on Spark:** OpenLineage 1.54 brought `jackson-module-blackbird` 2.18. The Beam fat jar bundles it, and it breaks on Spark 3.5 executors (jackson-databind 2.15), which shows up as a `serialVersionUID` mismatch. It is now pinned to 2.15.3.
- **Gremlin:** TinkerPop stays on the 3.7 line (3.7.7), because there is no stable JanusGraph release for 3.8 yet.
- **Tika:** `golden-apache-tika.csv` is updated in `mdi` and `transforms`. Tika 3.3.3 extracts three Office documents slightly differently: it keeps a soft hyphen, drops a trailing newline, and puts a space between hyperlinks.
- **GCP / BigQuery:** `protobuf-javalite` is excluded from `grpc-protobuf-lite` in hop-core. It clashed with `protobuf-java` in `lib/core` and caused `NoSuchMethodError: DurationProto.getDescriptor()`.
- **Mongo ITs on arm64:** the no-AVX `mongod` wrapper now only uses qemu on x86_64 hosts.
### New: GCS upload buffer size for Dataflow
Beam stages the fat jar in 1 MB chunks by default, one request per chunk, which caps staging at about 1 MB/s from a laptop. The Dataflow run configuration has a new **GCS upload buffer size (MB)** field. When it's empty, 128 MB is used. It is documented in the Dataflow run configuration page.
### Testing
- Unit tests: core, engine, Beam, Spark and all touched plugins.
- Integration tests that pass: `spark`, `spark-native`, `mongo`, both Tika tests, `beam_directrunner` 0006, `samples` 0001, and `transforms` 0102/0109/0111/0112.
- Not yet verified: the `protobuf-javalite` exclusion and the GCS upload buffer field. Both still need a GCP integration test run. The Google Sheets tests were skipped because there was no service-account key.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
- [ ] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`.
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [ ] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dependency upgrades and minor fixes, fixes #8966 (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva merged PR #8997:
URL: https://github.com/apache/hop/pull/8997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dependency upgrades and minor fixes, fixes #8966 (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on PR #8997:
URL: https://github.com/apache/hop/pull/8997#issuecomment-6270478743
Thanks @hansva, nice cleanup of lib/core. I checked that the jars plugins now take as `provided` (and the ones excluded from the Spark assembly) are all present in lib/core, and that pty4j 0.14 only uses Kotlin stdlib APIs available in the 1.9.25 we ship. Code reviewed, not built or run locally.
**Should fix**
- `BeamDataFlowPipelineRunConfiguration.getPipelineOptions()`: an empty field now means 128 MB instead of Beam's Dataflow default of 1 MB (`DataflowRunner.GCS_UPLOAD_BUFFER_SIZE_BYTES_DEFAULT`). The option is passed to the workers too and is the chunk size for every GCS write (file output, BigQuery file loads), so existing Dataflow configs would hold 128 MB per open writer. Suggest only setting it when the field is filled in, recommending 128 MB in the tooltip/docs, and capping the value (`mb * 1024 * 1024` overflows from 2048).
Minor: `ui/pom.xml` moves jediterm from 3.57 to 3.56. Intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org