You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fat-catTW (via GitHub)" <gi...@apache.org> on 2026/09/06 08:28:11 UTC
[PR] Fix Beam Dataflow deferral without launcher job id [airflow]
fat-catTW opened a new pull request, #72586:
URL: https://github.com/apache/airflow/pull/72586
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Why
Beam Dataflow operators relied on Beam launcher logs to discover the Dataflow job id before deferring. When those logs omit the id, the deferrable path could pass `None` into the Dataflow trigger and fail instead of using the existing job-name based wait path.
relates: #68279
## Solution
Try to resolve the Dataflow job id by job name while the Beam launcher is still running. If the id is found, deferrable execution proceeds with a valid trigger job id. If the id still cannot be resolved, fall back to the synchronous `wait_for_done` path instead of deferring with `None`.
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- `[X]` Yes (please specify the tool below)
Generated-by: [Codex] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Beam Dataflow deferral without launcher job id [airflow]
Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on PR #72586:
URL: https://github.com/apache/airflow/pull/72586#issuecomment-5571748682
Maybe, one way to validate the fix locally before #66952 is merged would be to create a temporary worktree based on #66952 and apply the fix on top of it?
This should allow the suspended Beam tests to run instead of being skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Beam Dataflow deferral without launcher job id [airflow]
Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on PR #72586:
URL: https://github.com/apache/airflow/pull/72586#issuecomment-5571445875
Thanks for pointing this out.
I confirmed that the Apache Beam provider tests are currently excluded from CI while the provider is suspended, and the related Google Dataflow test module is skipped as well.
I also reviewed #66952 and the existing fix in VladaZakharova/airflow#325. Although this PR takes a somewhat different approach, both changes address the same underlying issue.
Since the existing work started earlier and is already coordinated with the Beam provider unsuspension, I’m converting this PR to draft and will leave the issue to that work.
Please feel free to reference or reuse anything useful from this implementation, particularly the periodic job ID lookup while the launcher is still producing output, the synchronous fallback when no job ID can be resolved, and the additional Beam operator tests.
Thanks for the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Beam Dataflow deferral without launcher job id [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #72586:
URL: https://github.com/apache/airflow/pull/72586#issuecomment-5734286802
Closing as #72809 should solve the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Beam Dataflow deferral without launcher job id [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed pull request #72586: Fix Beam Dataflow deferral without launcher job id
URL: https://github.com/apache/airflow/pull/72586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org