You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2026/10/06 15:35:48 UTC

[PR] Cover long-running jobs in the deferrable BigQuery check system test [airflow]

shahar1 opened a new pull request, #74356:
URL: https://github.com/apache/airflow/pull/74356

   The deferrable `BigQueryValueCheckOperator` and `BigQueryIntervalCheckOperator` in the async BigQuery system test only ever ran trivial `COUNT(*)` queries, so their jobs were `DONE` before the trigger's first poll and the regression fixed in #74305 (#73981: triggers failed with `Job running` whenever the job was still `RUNNING`) could not be observed there.
   
   This extends `example_bigquery_queries_async.py` with a view that is slow on purpose and runs both deferrable checks against it:
   
   - `create_slow_view` creates a JavaScript UDF with a fixed 5e8-iteration loop and a view over the existing two-row table that evaluates it per row, so a query over the view stays `RUNNING` for tens of seconds. `CURRENT_TIMESTAMP()` in the view keeps BigQuery from serving repeated checks from the query cache.
   - `check_value_long_running` (`BigQueryValueCheckOperator`, deferrable) and `check_interval_long_running` (`BigQueryIntervalCheckOperator`, deferrable) run against the view. They are wired outside the `howto_operator_*` doc snippets so the documentation examples stay unchanged.
   
   Cost: on-demand billing charges for bytes scanned, not for UDF CPU time, and the view only reads the two-row table, so every check job is billed at BigQuery's 10 MB minimum.
   
   **Testing** against a real BigQuery project (on-demand, no reservations):
   
   - With the trigger from before #74305 (file checked out at `eda8563d98^`), `check_value_long_running` failed all three attempts with `AirflowException: Job running` (the check jobs took 15–45 s), reproducing #73981. In an earlier run with a shorter loop, `check_interval_long_running` also failed with the misreported `Job completed` message from the second bug in that issue.
   - With the current trigger, both long-running checks passed on the first attempt (check jobs took 8–48 s).
   - `check_openlineage_events` fails in my local pytest harness for both the modified Dag and the unmodified Dag (`Couldn't find any OpenLineage transport configuration` in task processes), so that task's result is unrelated to this change. The new tasks have no OpenLineage expectations.
   - Unit tests in `providers/google/tests/unit/google/cloud/triggers/test_bigquery.py` still cover the trigger logic; this PR only touches the system test.
   
   related: #73981
   related: #74305
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Claude Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cover long-running jobs in the deferrable BigQuery check system test [airflow]

Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 merged PR #74356:
URL: https://github.com/apache/airflow/pull/74356


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Cover long-running jobs in the deferrable BigQuery check system test [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #74356:
URL: https://github.com/apache/airflow/pull/74356#issuecomment-6019888432

   CC: @molcay @VladaZakharova


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org