You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "florianvazelle (via GitHub)" <gi...@apache.org> on 2026/04/28 15:30:56 UTC
[PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
florianvazelle opened a new pull request, #28059:
URL: https://github.com/apache/flink/pull/28059
## What is the purpose of the change
Based on https://github.com/apache/flink/pull/25541, upgrade Apache Beam dependency to 2.72.0.
The primary motivations are:
- Dependency management: Eases constraints on protobuf and other transitive dependencies to minimize conflicts across environments.
- Python compatibility: Enhances support for newer Python versions (like 3.12).
- Security: Fixes CVE-2024-52338 in pyarrow (critical severity, score 9.8) by upgrading to pyarrow >= 17.0.0 in Apache Beam >= 2.66.
---
## Brief change log
- Bump `apache-beam` version:
- from `>=2.54.0,<=2.61.0`
- to `>=2.54.0,<3.0.0`
---
## Verifying this change
This change is already covered by existing tests.
---
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **yes**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no**
- The S3 file system connector: **no**
---
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-5166824033
@dianfu @twalthr could you please take a look when you have time ? :pray:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4808162700
Also related to [FLINK-39031](https://issues.apache.org/jira/browse/FLINK-39031?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22pemja%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "khmelevskiy (via GitHub)" <gi...@apache.org>.
khmelevskiy commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4582144523
Hi @dianfu, sorry for the direct ping — I noticed you reviewed/merged previous PyFlink Beam dependency updates.
Would you or someone from the PyFlink side be able to take a look at this PR? We're hitting dependency conflicts around protobuf 6 through the current Beam constraint, so this bump would be very helpful for us in production.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4631828376
Up @twalthr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.75 [flink]
Posted by "dianfu (via GitHub)" <gi...@apache.org>.
dianfu commented on code in PR #28059:
URL: https://github.com/apache/flink/pull/28059#discussion_r3793441951
##########
flink-python/src/main/resources/META-INF/NOTICE:
##########
@@ -12,23 +12,25 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.21.3
- com.google.flatbuffers:flatbuffers-java:25.2.10
- joda-time:joda-time:2.5
+- io.opentelemetry:opentelemetry-api:1.57.0
+- io.opentelemetry:opentelemetry-context:1.57.0
Review Comment:
Dependency io.opentelemetry:opentelemetry-common:1.57.0 is not listed which caused the CI failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.75 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on code in PR #28059:
URL: https://github.com/apache/flink/pull/28059#discussion_r3794462412
##########
flink-python/src/main/resources/META-INF/NOTICE:
##########
@@ -12,23 +12,25 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.21.3
- com.google.flatbuffers:flatbuffers-java:25.2.10
- joda-time:joda-time:2.5
+- io.opentelemetry:opentelemetry-api:1.57.0
+- io.opentelemetry:opentelemetry-context:1.57.0
Review Comment:
Everything's done. Thanks for your reviews :pray:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4377253060
It appears that the only job failing in the Azure CI, is due to a network issue during the checkout step.
It's works on my side, in the Github CI https://github.com/florianvazelle/flink/actions/runs/25324694674.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4603905250
Hi @khmelevskiy, I'd also like to see this pull request merged.
If you have time, please feel free to take a look at my PR. I don't know if it will help the committers speed up the review process, but it's my first contribution to the project.
For another topic, do you know, if we don’t use the Protobuf format for the API table (https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/table/formats/protobuf/), we still need Protobuf? Or does Flink/Beam use it internally, or for something else?
If so, perhaps the dependency should be made optional. And a workaround, if you’re using uv, would be to use the [override dependencies](https://docs.astral.sh/uv/reference/settings/#override-dependencies). :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "florianvazelle (via GitHub)" <gi...@apache.org>.
florianvazelle commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4797095724
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-4336782327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8c7b7f06888ae3131347fab2e1ec1c22f3f14996",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c7b7f06888ae3131347fab2e1ec1c22f3f14996",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c7b7f06888ae3131347fab2e1ec1c22f3f14996 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "dianfu (via GitHub)" <gi...@apache.org>.
dianfu commented on PR #28059:
URL: https://github.com/apache/flink/pull/28059#issuecomment-5174469358
@florianvazelle Oh, sorry, I missed this PR. I will take a look at it these two days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.72 [flink]
Posted by "dianfu (via GitHub)" <gi...@apache.org>.
dianfu commented on code in PR #28059:
URL: https://github.com/apache/flink/pull/28059#discussion_r3763903487
##########
flink-python/README.md:
##########
@@ -39,7 +39,7 @@ python pyflink/gen_protos.py
```
PyFlink depends on the following libraries to execute the above script:
-1. grpcio-tools (>=1.29.0,<=1.71.0)
+1. grpcio-tools (>=1.29.0,<2)
Review Comment:
1.29.0 -> 1.33.1
##########
flink-python/setup.py:
##########
@@ -319,10 +319,10 @@ def extracted_output_files(base_dir, file_path, output_directory):
'pyflink.bin': ['*']}
install_requires = ['py4j==0.10.9.7', 'python-dateutil>=2.8.0,<3',
- 'apache-beam>=2.54.0,<=2.61.0',
+ 'apache-beam>=2.54.0,<3.0.0',
Review Comment:
We'd better limit the upper bound of apache-beam. There is no backward compatibility guarantee.
##########
flink-python/setup.py:
##########
@@ -319,10 +319,10 @@ def extracted_output_files(base_dir, file_path, output_directory):
'pyflink.bin': ['*']}
install_requires = ['py4j==0.10.9.7', 'python-dateutil>=2.8.0,<3',
- 'apache-beam>=2.54.0,<=2.61.0',
+ 'apache-beam>=2.54.0,<3.0.0',
Review Comment:
The protobuf>=6.31.1 requirement makes Beam 2.54–2.68 impossible to install: Beam 2.54 requires protobuf<4.26 and Beam 2.68 still requires protobuf<6. The effective minimum is Beam 2.69, so update this bound and the corresponding documentation, or relax the protobuf constraint
##########
flink-python/setup.py:
##########
@@ -319,10 +319,10 @@ def extracted_output_files(base_dir, file_path, output_directory):
'pyflink.bin': ['*']}
install_requires = ['py4j==0.10.9.7', 'python-dateutil>=2.8.0,<3',
- 'apache-beam>=2.54.0,<=2.61.0',
+ 'apache-beam>=2.54.0,<3.0.0',
'cloudpickle>=2.2.0', 'avro>=1.12.0',
'pytz>=2018.3', 'fastavro>=1.1.0,!=1.8.0', 'requests>=2.26.0',
- 'protobuf>=3.19.0',
+ 'protobuf>=6.31.1,<7.0.0.dev0',
Review Comment:
It accepts protobuf 6.31.1, but flink_fn_execution_pb2.py was regenerated with 6.33.5 and validates that exact minimum during import.
##########
flink-python/pyproject.toml:
##########
@@ -43,11 +43,11 @@ dev = [
"pytz>=2018.3",
"numpy>=1.22.4,<2.3.0",
"fastavro>=1.1.0,!=1.8.0",
- "grpcio>=1.29.0,<=1.71.0",
- "grpcio-tools>=1.29.0,<=1.71.0",
+ "grpcio>=1.33.1,<2",
+ "grpcio-tools>=1.33.1,<2",
Review Comment:
Could we pin grpcio-tools to 1.80.0 here, and use the same exact version in gen_protos.py and the README, instead of accepting a version range?
grpcio-tools is a code generator rather than an ordinary runtime dependency: it bundles a particular protoc version, and changing it can change the checked-in _pb2.py/_pb2.pyi files as well as their Protobuf runtime-version guard.
1.80.0 is the best fit for the compatibility contract currently declared by this PR. It is the latest grpcio-tools release that still supports Python 3.9, and it bundles libprotoc 31.1, which generates code requiring Protobuf 6.31.1—exactly matching the lower bound in setup.py. In contrast, 1.81.1 requires Python 3.10+ and generates code requiring Protobuf 6.33.5, while 1.82.1 and later require Protobuf 7 and therefore conflict with the current protobuf<7 constraint. Please pin grpcio-tools==1.80.0 and regenerate the checked-in protobuf files with that version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38469][python] Bump Apache Beam to 2.75 [flink]
Posted by "dianfu (via GitHub)" <gi...@apache.org>.
dianfu merged PR #28059:
URL: https://github.com/apache/flink/pull/28059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org