You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "ostinru (via GitHub)" <gi...@apache.org> on 2026/01/28 05:41:58 UTC
[I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
ostinru opened a new issue, #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42
Hi, fellas!
I'd like to bump minor versions of dependencies for all S3 and HDFS/Hive datasources.
At this moment I would like to:
* bump minor versions - get low hanging fruits: fix CVEs (if any) and get bugfixes.
* add support for zstd compression support
* test in manually & in CI
* probably fix couple tests from pxf_fdw's pool of broken tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3851167421
It seems that I have figured out how to solve HBASE-dependencies issue: https://github.com/apache/cloudberry-pxf/issues/51 . HBASE is not on my priority list, so I moved it to separate issue to keep this knowledge visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3845543023
## Apache ORC
Changelog 1.6.12 -> 1.7.x:
[ORC-757](https://issues.apache.org/jira/browse/ORC-757) Add Hashtable implementation for dictionary
Changelog 1.7.x -> 1.8.x:
[ORC-1136](https://issues.apache.org/jira/browse/ORC-1136) Optimize reads by combining multiple reads without significant separation into a single read
[ORC-1138](https://issues.apache.org/jira/browse/ORC-1138) Optimizes the read of streams in ORC by combining multiple nearby reads into a single read, optionally allowing the retention or drop of the extra bytes
[ORC-1172](https://issues.apache.org/jira/browse/ORC-1172) add row count limit config for one stripe
Changelog 1.8.x -> 1.9.x:
no important changes?
Changelog 1.9.x -> 2.0.x:
require Java 17
faster Zstd
Changelog 2.0.x -> 2.1.x:
require Java 17
Changelog 2.1.x -> 2.2.x:
require Java 17
[ORC-1903](https://issues.apache.org/jira/browse/ORC-1903) Support `Geometry` and `Geography` types
### Version Upgrade
At first glance it is reasonable to upgrade to 1.9.7. However it breaks something in HBASE support. PXF manages dependencies semi-manually (aka `transitive = false`). Either it pulls some fresh libraries OR it breaks with latest protobuf. Still investigating.
| orc-core | 1.6.13 | 1.7.11 | 1.8.10 | 1.9.8 |
|---|---|---|---|---|
|com.google.protobuf:protobuf-java| 2.5.0 | 2.5.0 | 3.17 | X |
|io.airlift:aircompressor| 0.21 | 0.21 | 0.21 | 0.27|
|org.apache.commons:commons-lang3| 3.12.0 | 3.12.0 | 3.12 | 3.12.0|
|org.apache.hadoop:hadoop-common| 2.2.0 | 2.2.0 | 2.7.3 | 2.7.3 (provided)|
|org.apache.hadoop:hadoop-hdfs| 2.2.0 | 2.2.0 | 2.7.3 | 2.7.3 (provided)|
|org.apache.hive:hive-storage-api| 2.7.3 | 2.8.1 | 2.8.1 | X|
|org.apache.orc:orc-shims| 1.6.14 | 1.7.11 | 1.8.10 | 1.9.8|
|org.jetbrains:annotations| 17 | 17 | 17 | 17|
|org.slf4j:slf4j-api| 1.7.5 | 1.7.33 | 1.7.36 | 2.0.7|
|org.threeten:threeten-extra| 1.5.0 | 1.5.0 | 1.7.1 | 1.7.1|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3809995688
Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3976430040
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3839986895
AVRO: https://github.com/apache/cloudberry-pxf/pull/45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3889164692
## Apache Parquet
Changes 1.11.1 -> 1.12.3 ( #56 )
* Bloom filter support
<In previous attempt Hbase blocked next updates. It was addressed in #62 >
Changes 1.12.x -> 1.13.x
* Support LZ4_RAW codec
Changes 1.13.x -> 1.14.x
* Java support for Arrow's float16
* Implement vectored IO in parquet file format
* More optimal memory usage in compression codecs
Changes 1.14.x -> 1.15.x
* nothing interesting, mostly dependencies update
Changes 1.15.x -> 1.16.x
* Remove hadoop-2 support
Changes 1.16.x -> 1.17.x
* Java 11+
### Version Upgrade
Dependency tree is small.
```
+--- org.apache.parquet:parquet-column:1.15.1
| +--- org.apache.parquet:parquet-common:1.15.1
| | +--- org.apache.parquet:parquet-format-structures:1.15.1
| +--- org.apache.parquet:parquet-encoding:1.15.1
| | +--- org.apache.parquet:parquet-common:1.15.1 (*)
+--- org.apache.parquet:parquet-hadoop:1.15.1
| +--- org.apache.parquet:parquet-column:1.15.1 (*)
| +--- org.apache.parquet:parquet-format-structures:1.15.1 (*)
| +--- org.apache.parquet:parquet-common:1.15.1 (*)
| +--- org.xerial.snappy:snappy-java:1.1.10.7
| +--- io.airlift:aircompressor:2.0.2
| +--- commons-pool:commons-pool:1.6
| +--- com.github.luben:zstd-jni:1.5.6-6
+--- org.apache.parquet:parquet-jackson:1.15.1
+--- org.apache.parquet:parquet-generator:1.15.1
+--- org.apache.parquet:parquet-pig:1.15.1
| +--- org.apache.parquet:parquet-column:1.15.1 (*)
| +--- org.apache.parquet:parquet-hadoop:1.15.1 (*)
| +--- org.apache.parquet:parquet-common:1.15.1 (*)
\--- org.apache.parquet:parquet-format:2.10.0
```
It has its own thrift library shaded. It doesn't depend on protobuf.
`paruet-hadoop` in fact expects that there is `hadoop-client`, `hadoop-common`, `hadoop-annotations` and `hadoop-mapreduce-client-core` provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3809595071
Regarding CVE, I have been researching for a while.
PXF depends on Spring Boot. Currently, the Spring Boot 3.4.x series has no vulnerabilities. However, their minimum required JDK version is 17.
Hadoop currently likely only supports up to JDK 11.
PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.
If we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3810142195
other cloudberry-contrib depends on jdk. see detail https://github.com/cloudberry-contrib/?q=&type=all&language=java&sort=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3809938489
Spring Boot has very short OSS support time (green boxes):
<img width="930" height="292" alt="Image" src="https://github.com/user-attachments/assets/325e40fe-100e-418e-b102-c549b0ef79ac" />
From `vmware-pxf` changelog it is clear that they are working with spring-boot-2.7.30.
And... from spring-boot changelog: it mostly dependencies updates.
And... almost all dependencies available (except enterprise spring-*).
So, we can live with spring-boot-2.7.x for a while with java 8..21.
However I would like to test it with java 17 (they said that every major JVM version gives several percent boost in performance) and with java 22+ (they said that JEP454 enables java to have faster compression/decompression libraries).
> PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.
>
> f we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.
Could you point on what in Cloudberry requires JVM?
Unless Cloudberry uses cloudberry-pxf java-libraries, it should be possible to run 2 different JVMs on the same machine.
> Hadoop currently likely only supports up to JDK 11.
I will check hadoop client-libraries.
PS: java / spring upgrade is out of scope for this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru closed issue #42: Dependencies update for S3/HDFS datasources
URL: https://github.com/apache/cloudberry-pxf/issues/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3810167032
Regarding the Spring issue, I suggest opening a new issue to discuss it ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3858757928
Parquet update merged: https://github.com/apache/cloudberry-pxf/pull/56
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "ostinru (via GitHub)" <gi...@apache.org>.
ostinru commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3810116152
> Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.
We have this page in our fork. There are a lot of notifications for:
* automation tests - there is no urgency to fix
* spring/tomcat - there are vulnerabilities, however it is hard to exploit when PXF is running on the same host as cloudberry (it is the only supported option).
* issues in other components - it can be dangerous (e.g. security issue in avro) - that is what I am want to fix in this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] Dependencies update for S3/HDFS datasources [cloudberry-pxf]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop commented on issue #42:
URL: https://github.com/apache/cloudberry-pxf/issues/42#issuecomment-3810083732
> Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.
404 not found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org