You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MartijnVisser (via GitHub)" <gi...@apache.org> on 2026/06/01 13:35:31 UTC
[PR] [FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop [flink]
MartijnVisser opened a new pull request, #28286:
URL: https://github.com/apache/flink/pull/28286
## What is the purpose of the change
`flink-gs-fs-hadoop` bundled `google-cloud-storage` 2.29.1, which throws a
`NullPointerException` instead of retrying certain GCS `503 Service Unavailable`
errors during resumable uploads. This breaks checkpointing for jobs that write to
`gs://` through a `RecoverableWriter` (e.g. the `FileSink`). The upstream fix is in
[googleapis/java-storage#2987](https://github.com/googleapis/java-storage/pull/2987),
which is included in newer releases of the library.
This PR takes over the stale #27679 (thanks @jonchase) and completes it by also
regenerating the bundled-dependency `NOTICE`, which is required because this module
shades all of its dependencies into the plugin jar.
## Brief change log
- Bump `google-cloud-storage` 2.29.1 -> 2.68.0 (latest stable) and the matching
grpc artifacts 1.59.1 -> 1.81.0 in `flink-gs-fs-hadoop/pom.xml`.
- Regenerate `META-INF/NOTICE` to match the bundled dependency set produced by the
shade plugin (verified to match exactly via `tools/ci/license_check.sh`).
- Add the bundled license file for the newly bundled `org.codehaus.woodstox:stax2-api`.
- Update the `google-cloud-storage` version link in the GCS filesystem docs
(English + Chinese).
## Verifying this change
This change is covered by the existing `flink-gs-fs-hadoop` tests (236 tests pass,
including `GSRecoverableWriterTest`, the committer/serializer tests, and the
`LocalStorageHelper`-based `GSBlobStorageImplTest`, confirming the pinned
`google-cloud-nio` test dependency stays compatible).
In addition, the license/NOTICE was validated locally with
`tools/ci/license_check.sh` (no severe issues; the `NOTICE` matches the 98 bundled
dependencies exactly) and the `dependency-convergence` and `ban-unsafe-jackson`
enforcers pass under `-Pcheck-convergence`.
Finally, the upgraded SDK was manually verified against a real GCS bucket: a
`RecoverableWriter` wrote a multi-chunk object, took a checkpoint via `persist()`,
recovered from that checkpoint via `recover()`, committed, and the committed object
was read back and asserted byte-for-byte (exactly-once), with no NPE/503 failure.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **yes** (upgrades
`google-cloud-storage` and grpc, with corresponding `NOTICE`/license updates)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components),
Checkpointing, Kubernetes/Yarn, ZooKeeper: **yes** (improves reliability of GCS
`RecoverableWriter` checkpoint/recovery by fixing 503 retry handling)
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #28286:
URL: https://github.com/apache/flink/pull/28286#discussion_r3336417001
##########
flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE:
##########
@@ -6,91 +6,117 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
+- com.fasterxml.jackson.core:jackson-annotations:2.21
- com.fasterxml.jackson.core:jackson-core:2.21.3
+- com.fasterxml.jackson.core:jackson-databind:2.21.3
+- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.21.3
+- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.21.3
+- com.fasterxml.woodstox:woodstox-core:7.0.0
- com.google.android:annotations:4.1.1.4
- com.google.api-client:google-api-client-jackson2:2.0.1
-- com.google.api-client:google-api-client:2.2.0
-- com.google.api.grpc:gapic-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:grpc-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:proto-google-cloud-monitoring-v3:1.64.0
-- com.google.api.grpc:proto-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:proto-google-common-protos:2.28.0
-- com.google.api.grpc:proto-google-iam-v1:1.23.0
+- com.google.api-client:google-api-client:2.7.2
+- com.google.api.grpc:gapic-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:grpc-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:proto-google-cloud-monitoring-v3:3.52.0
+- com.google.api.grpc:proto-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:proto-google-common-protos:2.71.0
+- com.google.api.grpc:proto-google-iam-v1:1.66.0
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
-- com.google.apis:google-api-services-storage:v1-rev20231028-2.0.0
-- com.google.auto.value:auto-value-annotations:1.10.4
+- com.google.apis:google-api-services-storage:v1-rev20260204-2.0.0
+- com.google.auto.value:auto-value-annotations:1.11.0
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18
- com.google.cloud.bigdataoss:gcsio:2.2.18
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18
- com.google.cloud.bigdataoss:util:2.2.18
-- com.google.cloud:google-cloud-core-grpc:2.27.0
-- com.google.cloud:google-cloud-core-http:2.27.0
-- com.google.cloud:google-cloud-core:2.27.0
-- com.google.cloud:google-cloud-monitoring:1.82.0
-- com.google.cloud:google-cloud-storage:2.29.1
-- com.google.code.gson:gson:2.10.1
+- com.google.cloud.opentelemetry:detector-resources-support:0.33.0
+- com.google.cloud.opentelemetry:exporter-metrics:0.33.0
+- com.google.cloud.opentelemetry:shared-resourcemapping:0.33.0
+- com.google.cloud:google-cloud-core-grpc:2.70.0
+- com.google.cloud:google-cloud-core-http:2.70.0
+- com.google.cloud:google-cloud-core:2.70.0
+- com.google.cloud:google-cloud-monitoring:3.52.0
+- com.google.cloud:google-cloud-storage:2.68.0
+- com.google.code.gson:gson:2.13.2
- com.google.flogger:flogger-system-backend:0.7.1
- com.google.flogger:flogger:0.7.1
- com.google.flogger:google-extensions:0.7.1
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
-- com.google.http-client:google-http-client-apache-v2:1.43.3
-- com.google.http-client:google-http-client-appengine:1.43.3
-- com.google.http-client:google-http-client-gson:1.43.3
-- com.google.http-client:google-http-client-jackson2:1.43.3
-- com.google.http-client:google-http-client:1.43.3
-- com.google.oauth-client:google-oauth-client:1.34.1
+- com.google.http-client:google-http-client-apache-v2:2.1.0
+- com.google.http-client:google-http-client-appengine:2.1.0
+- com.google.http-client:google-http-client-gson:2.1.0
+- com.google.http-client:google-http-client-jackson2:2.1.0
+- com.google.http-client:google-http-client:2.1.0
+- com.google.oauth-client:google-oauth-client:1.39.0
- com.lmax:disruptor:3.4.2
- commons-codec:commons-codec:1.15
-- io.grpc:grpc-alts:1.59.1
-- io.grpc:grpc-api:1.59.1
-- io.grpc:grpc-auth:1.59.1
-- io.grpc:grpc-census:1.59.1
-- io.grpc:grpc-context:1.59.1
-- io.grpc:grpc-core:1.59.1
-- io.grpc:grpc-googleapis:1.59.1
-- io.grpc:grpc-grpclb:1.59.1
-- io.grpc:grpc-inprocess:1.59.1
-- io.grpc:grpc-netty-shaded:1.59.1
-- io.grpc:grpc-protobuf-lite:1.59.1
-- io.grpc:grpc-protobuf:1.59.1
-- io.grpc:grpc-rls:1.59.1
-- io.grpc:grpc-services:1.59.1
-- io.grpc:grpc-stub:1.59.1
-- io.grpc:grpc-util:1.59.1
-- io.grpc:grpc-xds:1.59.1
+- io.grpc:grpc-alts:1.81.0
+- io.grpc:grpc-api:1.81.0
+- io.grpc:grpc-auth:1.81.0
+- io.grpc:grpc-census:1.81.0
+- io.grpc:grpc-context:1.81.0
+- io.grpc:grpc-core:1.81.0
+- io.grpc:grpc-googleapis:1.81.0
+- io.grpc:grpc-grpclb:1.81.0
+- io.grpc:grpc-inprocess:1.81.0
+- io.grpc:grpc-netty-shaded:1.81.0
+- io.grpc:grpc-opentelemetry:1.81.0
+- io.grpc:grpc-protobuf-lite:1.81.0
+- io.grpc:grpc-protobuf:1.81.0
+- io.grpc:grpc-rls:1.81.0
+- io.grpc:grpc-services:1.81.0
+- io.grpc:grpc-stub:1.81.0
+- io.grpc:grpc-util:1.81.0
+- io.grpc:grpc-xds:1.81.0
- io.opencensus:opencensus-api:0.31.1
- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
- io.opencensus:opencensus-contrib-http-util:0.31.1
- io.opencensus:opencensus-contrib-resource-util:0.31.0
- io.opencensus:opencensus-exporter-metrics-util:0.31.0
- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.0
-- io.opencensus:opencensus-impl:0.31.0
- io.opencensus:opencensus-impl-core:0.31.0
-- io.opencensus:opencensus-proto:0.2.0
-- io.perfmark:perfmark-api:0.26.0
+- io.opencensus:opencensus-impl:0.31.0
+- io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
+- io.opentelemetry.semconv:opentelemetry-semconv:1.29.0-alpha
+- io.opentelemetry:opentelemetry-api:1.57.0
+- io.opentelemetry:opentelemetry-common:1.57.0
+- io.opentelemetry:opentelemetry-context:1.57.0
+- io.opentelemetry:opentelemetry-sdk-common:1.57.0
+- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.57.0
+- io.opentelemetry:opentelemetry-sdk-logs:1.57.0
+- io.opentelemetry:opentelemetry-sdk-metrics:1.57.0
+- io.opentelemetry:opentelemetry-sdk-trace:1.57.0
+- io.opentelemetry:opentelemetry-sdk:1.57.0
+- io.perfmark:perfmark-api:0.27.0
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.conscrypt:conscrypt-openjdk-uber:2.5.2
+- org.jspecify:jspecify:1.0.0
Review Comment:
Excluded it from the shaded jar and dropped it from the NOTICE, consistent with the other annotation-only dependencies already excluded in this module (checker-qual, error_prone_annotations, j2objc-annotations).
One nuance: jspecify's annotations are actually `@Retention(RUNTIME)` rather than CLASS/SOURCE, but they're static-analysis nullness markers that nothing in the gcs-connector / google-cloud-storage stack reads reflectively at runtime, so excluding them is safe. Verified the module still builds, the unit tests pass, and the newly added real-GCS RecoverableWriter/FileSystem ITCases pass against a live bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28286:
URL: https://github.com/apache/flink/pull/28286#issuecomment-4593155376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0d2fd3079e465f57a6967c63b530852977f41474",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d2fd3079e465f57a6967c63b530852977f41474",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d2fd3079e465f57a6967c63b530852977f41474 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28286:
URL: https://github.com/apache/flink/pull/28286#discussion_r3335815161
##########
flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE:
##########
@@ -6,91 +6,117 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
+- com.fasterxml.jackson.core:jackson-annotations:2.21
- com.fasterxml.jackson.core:jackson-core:2.21.3
+- com.fasterxml.jackson.core:jackson-databind:2.21.3
+- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.21.3
+- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.21.3
+- com.fasterxml.woodstox:woodstox-core:7.0.0
- com.google.android:annotations:4.1.1.4
- com.google.api-client:google-api-client-jackson2:2.0.1
-- com.google.api-client:google-api-client:2.2.0
-- com.google.api.grpc:gapic-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:grpc-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:proto-google-cloud-monitoring-v3:1.64.0
-- com.google.api.grpc:proto-google-cloud-storage-v2:2.29.1-alpha
-- com.google.api.grpc:proto-google-common-protos:2.28.0
-- com.google.api.grpc:proto-google-iam-v1:1.23.0
+- com.google.api-client:google-api-client:2.7.2
+- com.google.api.grpc:gapic-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:grpc-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:proto-google-cloud-monitoring-v3:3.52.0
+- com.google.api.grpc:proto-google-cloud-storage-v2:2.68.0
+- com.google.api.grpc:proto-google-common-protos:2.71.0
+- com.google.api.grpc:proto-google-iam-v1:1.66.0
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
-- com.google.apis:google-api-services-storage:v1-rev20231028-2.0.0
-- com.google.auto.value:auto-value-annotations:1.10.4
+- com.google.apis:google-api-services-storage:v1-rev20260204-2.0.0
+- com.google.auto.value:auto-value-annotations:1.11.0
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18
- com.google.cloud.bigdataoss:gcsio:2.2.18
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18
- com.google.cloud.bigdataoss:util:2.2.18
-- com.google.cloud:google-cloud-core-grpc:2.27.0
-- com.google.cloud:google-cloud-core-http:2.27.0
-- com.google.cloud:google-cloud-core:2.27.0
-- com.google.cloud:google-cloud-monitoring:1.82.0
-- com.google.cloud:google-cloud-storage:2.29.1
-- com.google.code.gson:gson:2.10.1
+- com.google.cloud.opentelemetry:detector-resources-support:0.33.0
+- com.google.cloud.opentelemetry:exporter-metrics:0.33.0
+- com.google.cloud.opentelemetry:shared-resourcemapping:0.33.0
+- com.google.cloud:google-cloud-core-grpc:2.70.0
+- com.google.cloud:google-cloud-core-http:2.70.0
+- com.google.cloud:google-cloud-core:2.70.0
+- com.google.cloud:google-cloud-monitoring:3.52.0
+- com.google.cloud:google-cloud-storage:2.68.0
+- com.google.code.gson:gson:2.13.2
- com.google.flogger:flogger-system-backend:0.7.1
- com.google.flogger:flogger:0.7.1
- com.google.flogger:google-extensions:0.7.1
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
-- com.google.http-client:google-http-client-apache-v2:1.43.3
-- com.google.http-client:google-http-client-appengine:1.43.3
-- com.google.http-client:google-http-client-gson:1.43.3
-- com.google.http-client:google-http-client-jackson2:1.43.3
-- com.google.http-client:google-http-client:1.43.3
-- com.google.oauth-client:google-oauth-client:1.34.1
+- com.google.http-client:google-http-client-apache-v2:2.1.0
+- com.google.http-client:google-http-client-appengine:2.1.0
+- com.google.http-client:google-http-client-gson:2.1.0
+- com.google.http-client:google-http-client-jackson2:2.1.0
+- com.google.http-client:google-http-client:2.1.0
+- com.google.oauth-client:google-oauth-client:1.39.0
- com.lmax:disruptor:3.4.2
- commons-codec:commons-codec:1.15
-- io.grpc:grpc-alts:1.59.1
-- io.grpc:grpc-api:1.59.1
-- io.grpc:grpc-auth:1.59.1
-- io.grpc:grpc-census:1.59.1
-- io.grpc:grpc-context:1.59.1
-- io.grpc:grpc-core:1.59.1
-- io.grpc:grpc-googleapis:1.59.1
-- io.grpc:grpc-grpclb:1.59.1
-- io.grpc:grpc-inprocess:1.59.1
-- io.grpc:grpc-netty-shaded:1.59.1
-- io.grpc:grpc-protobuf-lite:1.59.1
-- io.grpc:grpc-protobuf:1.59.1
-- io.grpc:grpc-rls:1.59.1
-- io.grpc:grpc-services:1.59.1
-- io.grpc:grpc-stub:1.59.1
-- io.grpc:grpc-util:1.59.1
-- io.grpc:grpc-xds:1.59.1
+- io.grpc:grpc-alts:1.81.0
+- io.grpc:grpc-api:1.81.0
+- io.grpc:grpc-auth:1.81.0
+- io.grpc:grpc-census:1.81.0
+- io.grpc:grpc-context:1.81.0
+- io.grpc:grpc-core:1.81.0
+- io.grpc:grpc-googleapis:1.81.0
+- io.grpc:grpc-grpclb:1.81.0
+- io.grpc:grpc-inprocess:1.81.0
+- io.grpc:grpc-netty-shaded:1.81.0
+- io.grpc:grpc-opentelemetry:1.81.0
+- io.grpc:grpc-protobuf-lite:1.81.0
+- io.grpc:grpc-protobuf:1.81.0
+- io.grpc:grpc-rls:1.81.0
+- io.grpc:grpc-services:1.81.0
+- io.grpc:grpc-stub:1.81.0
+- io.grpc:grpc-util:1.81.0
+- io.grpc:grpc-xds:1.81.0
- io.opencensus:opencensus-api:0.31.1
- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
- io.opencensus:opencensus-contrib-http-util:0.31.1
- io.opencensus:opencensus-contrib-resource-util:0.31.0
- io.opencensus:opencensus-exporter-metrics-util:0.31.0
- io.opencensus:opencensus-exporter-stats-stackdriver:0.31.0
-- io.opencensus:opencensus-impl:0.31.0
- io.opencensus:opencensus-impl-core:0.31.0
-- io.opencensus:opencensus-proto:0.2.0
-- io.perfmark:perfmark-api:0.26.0
+- io.opencensus:opencensus-impl:0.31.0
+- io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
+- io.opentelemetry.semconv:opentelemetry-semconv:1.29.0-alpha
+- io.opentelemetry:opentelemetry-api:1.57.0
+- io.opentelemetry:opentelemetry-common:1.57.0
+- io.opentelemetry:opentelemetry-context:1.57.0
+- io.opentelemetry:opentelemetry-sdk-common:1.57.0
+- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.57.0
+- io.opentelemetry:opentelemetry-sdk-logs:1.57.0
+- io.opentelemetry:opentelemetry-sdk-metrics:1.57.0
+- io.opentelemetry:opentelemetry-sdk-trace:1.57.0
+- io.opentelemetry:opentelemetry-sdk:1.57.0
+- io.perfmark:perfmark-api:0.27.0
- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.conscrypt:conscrypt-openjdk-uber:2.5.2
+- org.jspecify:jspecify:1.0.0
Review Comment:
do we need it?
AFAIK it is only annotations for compile time checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #28286:
URL: https://github.com/apache/flink/pull/28286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org