You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "olegkachur-e (via GitHub)" <gi...@apache.org> on 2026/05/14 14:46:45 UTC
[PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
olegkachur-e opened a new pull request, #66952:
URL: https://github.com/apache/airflow/pull/66952
This reverts commit 917abea0d2b2d219e4158fe4f31280378a36af4c.
- Remove hacks regarding the beam provider suspension, as they are not relevant anymore.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4929269487
> We will not add explicit dependency on a pre-relesse/beta package.
We actually don't depend on this dep directly in Beam either, i am not sure how it came into the picture in this Airflow change, presumably to help with the dependency resolution, and a suggestion is to add constraints on envoy-data-plane instead if you must appease the resolver for some reason.
> production grade the mantainers can just remove it from PyPi without further notice
we are going into hypotheticals here, while there are real adverse effects for users of Airflow and Beam due to suspended provider that I'd rather focus on. Left a suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4927313175
> Again, Beam needs to explain why they can't do out of release schedule to mitigate. Airflow is doing ad hoc releases when needed.
An adhoc release would require you to set the upper bound to `apache_beam>= 2.75.1` . Taking the best interests of Airflow users in mind, this might not be a great solution since if Beam 2.75.0 happens to be buggy, your users won't have an option to downgrade for a while.
I agree that depending on 'beta' doesn't have great optics but beyond optics per se, i don't see an actual risk (i.e. vulnerability or introducing bugs). Arguably that was Beam's fault, not Airflow's fault and you are not adding this dependency intentionally.
would it be ok for airflow to add a constraint `envoy_data_plane==1.0.3` instead of `betterproto==2.0.0b6` if you are concerned with mentioning a beta dependency in your codebase?
Beam can do out-of-calendar releases but this is a nontrivial amount of work and the bar for that is high (for example an urgent dataloss issue or smth like log4j vulnerability that everyone was scrambilng to mitigate asap). During the entire project lifetime that happened perhaps 2 or 3 times .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4913193162
@shahar1 awesome! But we still need to wait for apache beam to be released, so we can remove constraint of `betterproto==2.0.0b6` right?
@eladkal thank you for the review, I'll adjust the commit message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3474031310
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I am not a big fan of this idea either.
The reason why I decided to add it was in that this beta version exists from 2023 and I am not sure that it will be release in the future. One more point for me to added it was that I noticed that Apache Beam SDK use this beta version in there [generated requirements](https://github.com/apache/beam/blob/v2.74.0/sdks/python/container/py311/base_image_requirements.txt#L35) for containers. And I had idea that if they already had hardcoded it in `apache-beam` requirements, from which our provider already depends, maybe it makes sense temporary hardcoded it in provider as well for fixing constrains problem and unsuspend provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3469788339
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I am not a fan of this.
this is a beta version from 2023. This is not a case where in a few days/weeks things will change and a stable version would be released, is it?
I am not in the details of this so I will leave @potiuk the call here but from my point of view we should not deliever stable release with depenedency on beta packages.
Who gurentees that tommorow the author won't remove them from PyPi with a thought that there won't be any production impact on anyone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4794905420
I commented in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3474124566
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
We can't tell beam what to so from thier side but Airflow should not do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3521161970
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
Created this PR in beam to try and tackle this issue:
https://github.com/apache/beam/pull/39213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3520031869
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
This exact-prerelease pin is a workaround for the `envoy-data-plane` 1.0.3 constraint, so per the project's workaround convention it needs a GitHub tracking issue — linked by full URL here at the pin site (not a bare `#NNNNN`) and referenced in the PR body — capturing the follow-up: getting apache-beam onto `envoy-data-plane>=2` / `betterproto2` so this `==2.0.0b6` pin can be removed. Without that, an exact prerelease pin tends to quietly become permanent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3629528021
##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -978,12 +978,13 @@ def check_doc_files(yaml_files: dict[str, dict]) -> tuple[int, int]:
if f.name != "index.rst" and "_partials" not in f.parts and f.parts[2] == "docs"
}
- expected_doc_urls = {
- doc_url
- for doc_url in expected_doc_urls
- for suspend_provider in suspended_providers
- if suspend_provider not in doc_url
- }
+ if suspended_providers:
Review Comment:
@gopidesupavan this `if-clause` is needed because in case when `suspended_providers` is empty. This code:
```
expected_doc_urls = {
doc_url
for doc_url in expected_doc_urls
for suspend_provider in suspended_providers
if suspend_provider not in doc_url
}
```
rewrite `expected_doc_urls` and assign `{}` as a value and as a result broke CI tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4916759845
Could we possibly reevaluate the decision to block on Beam 2.76.0? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4913487103
> UPD: the beam 2.75 was [released](https://pypi.org/project/apache-beam/2.75.0/) a couple hours ago, but unfortunately @shahar1 fix seems to be not included into this release:
>
> ```
> pip list | grep beam
> apache-beam 2.75.0
> pip list | grep betterproto
> betterproto 2.0.0b6
> ```
By "> 2.75" I meant "strictly larger than", as I was aware of the ongoing 2.75 release process :)
So let's wait for the next one, it should be probably in the next month or two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4928418803
to clarify: what is the risk for Airflow you'd like address ? If we look at the lock file, there are dependencies that are even older than betterproto. Is the word 'beta' in the name the only risk here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3554630320
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I am not sure why we need this particular dep here -- is it because it has a python version specifier?
Consider the following instead, which matches the beam constraint.
```suggestion
"envoy-data-plane>=1.0.3,<2; python_version >= '3.11'",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4919041705
> Given the risk is known, bounded, and time-limited, I think unsuspending now is reasonable rather than keeping users on a suspended provider for another release cycle
I don't understand. what is the request?
Is the request to ship beam provider with a beta dependecy?
Before I write my thoughts there should be an explanation from beam mantainers why can't they just do another release? Shifting the risk to Airflow, however minimal, is a big ask. This goes way beyond just Beam. The desicion that will be taken here will be used as precedence for all the 100+ providers we have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4905870798
@olegkachur-e
1. Please resolve conflicts.
2. My [PR](https://github.com/apache/beam/pull/39213) in Apache beam has been merged, so we should be able to unsuspend it in version > 2.75.
Let's keep watching until it is shipped, and then we'll be able to unsuspend it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4928718542
> Is the word 'beta' in the name the only risk here?
It's not a word. It is a declaration.
https://pypi.org/project/betterproto/2.0.0b6/
The mantainers of this lib marked it as not suitable for production. No new releases since 2024.
Technically, since its not production grade the mantainers can just remove it from PyPi without further notice. It is a huge risk to take dependencies on such releases. Beam can do it from your side. Airflow isn't likely to do it. We can bring beam to dependencies but that is where the line is drawn. We will not add explicit dependency on a pre-relesse/beta package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4916744391
> 2. My [PR](https://github.com/apache/beam/pull/39213) in Apache beam has been merged, so we should be able to unsuspend it in version > 2.75.
> Let's keep watching until it is shipped, and then we'll be able to unsuspend it.
@shahar1 thanks for doing that.
Regarding betterproto dependency:
* the functionality where envoy-data-plane is used is an opt-in functionality of Beam that doesn't have significant usage atm
* While Betterproto doesn't have any recent releases: https://pypi.org/project/betterproto/#history, is there any concern as to this dependency having a vulnerability? It seems like we flagged it primarily due to beta in the naming, and then noticed that the codebase is old. Even if a project has recent releases, it can have codepaths that haven't been touched for years. Do we have any reason to be concerned beside the name ?
* As mentioned, Beam will drop the dependency on the envoy-data-plane in the next release, but using older version of Beam does more harm in this case to airflow users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4918945757
> hi all, Beam committer here.
>
> > 2. My [PR](https://github.com/apache/beam/pull/39213) in Apache beam has been merged, so we should be able to unsuspend it in version > 2.75.
> > Let's keep watching until it is shipped, and then we'll be able to unsuspend it.
>
> @shahar1 thanks for doing that.
>
> Regarding betterproto dependency:
>
> * the functionality where envoy-data-plane is used is an opt-in functionality of Beam that doesn't have significant usage atm
> * While Betterproto doesn't have any recent releases: https://pypi.org/project/betterproto/#history, is there any concern as to this dependency having a vulnerability? It seems like we flagged it primarily due to beta in the naming, and then noticed that the codebase is old. Even if a project has recent releases, it can have codepaths that haven't been touched for years. Do we have any reason to be concerned beside the name ?
> * As mentioned, Beam will drop the dependency on the envoy-data-plane in the next release, but using older version of Beam in my opinion causes more inconvenience in this case to airflow users than pulling an old dependency .
Thanks for your comment!
I reevaluated this:
- The original suspension blocker (Beam's grpcio<1.66 pin) is resolved as of Beam 2.72.
- There are no known vulnerabilities in Beam 2.72–2.75, nor in `betterproto==2.0.0b6` or `envoy-data-plane==1.0.3`.
- `envoy-data-plane==1.0.3` is part of an opt-in Beam feature with little usage.
- The dependency is already removed upstream (apache/beam#39213) and ships in Beam 2.76, expected in ~1–2 months.
Given the risk is known, bounded, and time-limited, I think unsuspending now is reasonable rather than keeping users on a suspended provider for another release cycle. We already accept unmaintained transitive dependencies elsewhere in the tree on a case-by-case basis; this one at least has a concrete removal date. Of course, if there's a vunlerability found and 2.76 is yet to be released - we could resume the suspension.
@potiuk @eladkal WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3554630320
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I am not sure why we need this particular dep here -- is it because beam has a python version constraint on envoy-data-plane?
If so, consider the following instead, which matches the beam constraint.
```suggestion
"envoy-data-plane>=1.0.3,<2; python_version >= '3.11'",
```
what happens if we don't add any additional constraints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4877692673
Please also amend commit message to a meaningful one. Revert old commit is not very informative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3537957510
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
Merged :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-5045419456
Hello @gopidesupavan @potiuk ,
This PR is in charge of me not in @olegkachur-e. I was on vacation last two weeks and could not respond on comments and fixed merge conflicts in the current PR. Yesterday I fixed merge conflicts and as I see your guys in the evening added additional changes to the code. Could you please clarify for me should I continue to work on this PR or it is not on my duty anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3626390027
##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -978,12 +978,13 @@ def check_doc_files(yaml_files: dict[str, dict]) -> tuple[int, int]:
if f.name != "index.rst" and "_partials" not in f.parts and f.parts[2] == "docs"
}
- expected_doc_urls = {
- doc_url
- for doc_url in expected_doc_urls
- for suspend_provider in suspended_providers
- if suspend_provider not in doc_url
- }
+ if suspended_providers:
Review Comment:
ah okay ignore this comment ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-5593189134
I think I fixed the issue - rebased and provided a fixup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #66952:
URL: https://github.com/apache/airflow/pull/66952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4592364949
@olegkachur-e A few things need addressing before review — see our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
- :x: **Unit tests**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/09_testing.rst).
- :x: **Provider tests**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/12_provider_distributions.rst).
- :x: **Other failing CI checks**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
- :x: **Image build**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
No rush.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4927832253
> your users won't have an option to downgrade for a while.
I'd argue they are beam users as well :)
With providers we relay on upstream fixes. If beam is not willing to cut a release as was suggested here I'd suggest to work with @shahar1 on a proposal that doesn't put the risks on Airflow. If it has merit i will not object. by merit I mean actually solve our concerns, trying to create some cosmetic appears as if issue was addressed while not is not a fix. Again, I am only one voice here. There are many PMCs in the project who might think differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3626381475
##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -978,12 +978,13 @@ def check_doc_files(yaml_files: dict[str, dict]) -> tuple[int, int]:
if f.name != "index.rst" and "_partials" not in f.parts and f.parts[2] == "docs"
}
- expected_doc_urls = {
- doc_url
- for doc_url in expected_doc_urls
- for suspend_provider in suspended_providers
- if suspend_provider not in doc_url
- }
+ if suspended_providers:
Review Comment:
We shouldn’t remove these. They were flagged when we suspended the provider, and these helpers will be useful again if we need to suspend any other provider in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3554630320
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I am not sure why we need this particular dep here -- is it because beam has a python version constraint on envoy-data-plane?
If so, consider the following instead, which matches the beam constraint.
```suggestion
"envoy-data-plane>=1.0.3,<2; python_version >= '3.11'",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-5090619585
> Hello @gopidesupavan @potiuk , This PR is in charge of me not in @olegkachur-e. I was on vacation last two weeks and could not respond on comments and fixed merge conflicts in the current PR. Yesterday I fixed merge conflicts and as I see your guys in the evening added additional changes to the code. Could you please clarify for me should I continue to work on this PR or it is not on my duty anymore?
Hello @MaksYermak - sorry for not responding before - I've been sick/out of action for the last week or so and slowly getting back to regular involvement.
I tried to see - similarly as @gopidesupavan to fix the dependency issue, but as @eladkal noted - fixing betterproto to 6 years old pre-release version is a bad idea. Really the issue is on beam side not ours, so it's up on them to fix it.
Luckily - they already did - it just did not make it into 2.75.0 release in July and it is planned to be released at the beginning of August. And this is the right way of solving the problem - and opens the way to un-suspend beam provider - making any workarounds on our side is just not something we should accept - especially that there is a fix in sight.
You can read more details in here, where I wrote message to the dev team of Apache Beam:
https://lists.apache.org/thread/lnm2gszq3lvqxgz70rms7zqmr757w5ol
So - at this stage, what we need to do is to wait until 2.76.0 is released by Apache beam. Once this is done (+3 days cooldown on our side) - you should just rebase/resolve issues, bump min version for it pyproject.toml of the beam provider and all issues **should** be solved.
I think a good idea for you might be to respond to that message if you feel that Beam team should speed it up a bit maybe - explaining how much needed it is by the composer team - and Google customers. Yes. I know it's frustrating, but it's really not something we should fix or workaround here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3470378494
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
Not a big fan either.
But I think thee newer version of envoy-data-plane has been questioned by others for example here: https://github.com/apache/beam/issues/37854
And I see that this is what latest beam has:
```
envoy-data-plane<2,>=1.0.3; python_version >= "3.11"
```
And the 2.1.0 version of envoy-data-plane uses already betterproto2 - which is a maintained replacement of betterproto.
So maybe we can ask `apache-beam` to upgrade to the `envoy-data-plane>2` ?
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
Not a big fan either.
But I think thee newer version of envoy-data-plane has been questioned by others for example here: https://github.com/apache/beam/issues/37854
And I see that this is what latest beam has:
```
envoy-data-plane<2,>=1.0.3; python_version >= "3.11"
```
And the 2.1.0 version of envoy-data-plane uses already betterproto2 - which is a maintained replacement of betterproto.
So maybe we can ask `apache-beam` to upgrade to the `envoy-data-plane>=2` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3471217804
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
I commented in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4848944111
<!-- uv-lock-rebase-notice -->
`uv.lock` on `main` just moved via [#68933](https://github.com/apache/airflow/pull/68933) ("[main] Upgrade important CI environment"), commit [`e0a1726`](https://github.com/apache/airflow/commit/e0a1726aa586800ec65e2626b4d110b83ecc88fa) and this PR currently conflicts.
Quickest fix:
```bash
git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease
```
_Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future `uv.lock` bumps._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4913463228
UPD: the beam 2.75 was [released](https://pypi.org/project/apache-beam/2.75.0/) a couple hours ago, but unfortunately @shahar1 fix seems to be not included into this release:
```
pip list | grep beam
apache-beam 2.75.0
pip list | grep betterproto
betterproto 2.0.0b6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #66952:
URL: https://github.com/apache/airflow/pull/66952#issuecomment-4919105277
> The dependency is already removed upstream ([Python] Drop envoy-data-plane/betterproto dependency from the Python SDK beam#39213) and ships in Beam 2.76, expected in ~1–2 months.
Again, Beam needs to explain why they can't do out of release schedule to mitigate. Airflow is doing ad hoc releases when needed. I assume beam can do that too.
To clarify, this is not a desicion we can take here. This must go to mailing list for visability. For exmaple, who decide what is resonable time? Is 1 month resonable? 3? What hapoens if beam break the commitment? Sometimes the things we expect don't work as much. We know this very well. Releases are delayed. Releases are yanked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #66952:
URL: https://github.com/apache/airflow/pull/66952#discussion_r3474124566
##########
providers/apache/beam/pyproject.toml:
##########
@@ -61,7 +61,11 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- "apache-beam>=2.69.0",
+ "apache-beam>=2.72.0",
+ # For Python>=3.11, the apache-beam package depends on envoy-data-plane.
+ # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
+ # Python versions, and it explicitly requires a prerelease version of betterproto.
+ "betterproto==2.0.0b6; python_version >= '3.11'",
Review Comment:
We can't tell beam what to do from thier side but Airflow should not do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org