You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jobs@airflow.apache.org by GitBox <gi...@apache.org> on 2026/06/28 07:21:04 UTC
[GH] (airflow/feature/68279-fix-deferrable-beam-dataflow-job-id): Workflow run "Tests (AMD)" failed!
The GitHub Actions job "Tests (AMD)" on airflow.git/feature/68279-fix-deferrable-beam-dataflow-job-id has failed.
Run started by GitHub user gingeekrishna (triggered by henry3260).
Head commit for run:
d8de40e5321202d04d2a45a39fb951cf3f4a1a99 / gingeekrishna <gi...@gmail.com>
Fix deferrable Beam Dataflow operator failing with 400 when job ID is missing from stdout
When the Dataflow launcher process runs with WARNING log level (the default),
it does not emit the "Created job with id" line that the Beam operator parses
to capture the Dataflow job ID. This left dataflow_job_id as None, causing the
deferrable trigger to fail with "400 Request must contain a job and project id".
Fix by adding a periodic_callback parameter to run_beam_command() that is
invoked roughly every 5 seconds while the launcher subprocess is running. The
deferrable Beam operators now pass a callback that polls DataflowHook.fetch_job_id_by_name()
to resolve the job ID by name. Once the ID is set, the stdout-reading loop exits
early so the operator can truly defer, freeing the Airflow worker while the
Dataflow job continues running on Google Cloud.
Fixes #68279
Report URL: https://github.com/apache/airflow/actions/runs/28313367881
With regards,
GitHub Actions via GitBox
---------------------------------------------------------------------
To unsubscribe, e-mail: jobs-unsubscribe@airflow.apache.org
For additional commands, e-mail: jobs-help@airflow.apache.org