You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "olegkachur-e (via GitHub)" <gi...@apache.org> on 2026/05/05 09:35:33 UTC

[PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

olegkachur-e opened a new pull request, #66404:
URL: https://github.com/apache/airflow/pull/66404

   - Fix deferrable mode for bigquery operators working with
   private cloud. The glocud-aio lib used does not yet support
   non-default universes, so we handle those special cases. 
   
   <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #66404:
URL: https://github.com/apache/airflow/pull/66404


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189988141


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   Still this type inconsistency a bit bugs me (could be easily solved by a simple input validation) - I'll appreciate if you could fix it, if you don't have the time it could wait for later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189810453


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   the type of `self.location` is `str | None`, `get_query_results` expects `str`



##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Non-default universe support seem missing here (`is_default_universe` etc.)
   Is it on purpose, or just a slip?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189880798


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   Good point, I'll update this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189901495


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   But this code will run only in Private Cloud, so the location should always be provided, otherwise it will not work anyway.  There no guaranteed default locations as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3191169436


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   I agree with your concerns, during test run there no issues with check, so I remove this changes from it, will take a look and include changes in the next wave.
   
   As for the CI taking a look.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on PR #66404:
URL: https://github.com/apache/airflow/pull/66404#issuecomment-4382842925

   > Overall LGTM. @olegkachur-e please address my last comment (not blocking, but might need to be revisited)
   
   Thank you for the thoroughful review! 
   
   Comments are addressed, waiting for the CI (local tests were green)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189880798


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   ~~Good point, I'll update this part.~~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189865893


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   I've got error on a custom universe, so adding location fix it, since it's valid parameter for default GCP it should not hurt.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on PR #66404:
URL: https://github.com/apache/airflow/pull/66404#issuecomment-4378100263

   This changes relies on the https://github.com/apache/airflow/pull/66159 base, should be merged after.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3190023479


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   `BigQueryCheckTrigger.run()` calls `hook.get_job_output()` on success, which is the gcloud-aio path. That call will fail on non-default universe domains (regardless of the location fix to `get_job_status`).
   
   Isn't it a problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #66404:
URL: https://github.com/apache/airflow/pull/66404#issuecomment-4378156797

   Hi @shahar1 !
   This PR is a part of new upcoming change in google provider and i was hoping it will be released in the nearest release (5th May). Can you please take a look and maybe merge it, so we can try to aim till the release day? :)
   thank you in advance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3190834492


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Unersolving as it might be related to current CI failure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189839052


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Non-default universe support seems missing here (`is_default_universe` etc.)
   Is it on purpose, or just a slip?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189988141


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   Still this type inconsistency a bit bugs me (could be easily solved by a simple type validation) - I'll appreciate if you could fix it, if you don't have the time it could wait for later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3190016175


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   Sure, I'll add the validation to be more explicit about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3190023479


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   `BigQueryCheckTrigger.run()` calls `hook.get_job_output()` on success, which is the gcloud-aio path. That call will fail on non-default universe domains, (regardless of the location fix to `get_job_status`).
   
   Isn't it a problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3190023479


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   `BigQueryCheckTrigger.run()` calls `hook.get_job_output()` on success, which is the gcloud-aio path. That call will fail on non-default universe domains, regardless of the location fix to `get_job_status`.
   
   Isn't it a problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189955356


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Oh, got your point. The get_job_status (https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2106)  calls the https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2098 under the hood which is sync_to_async wrapped call (not the glcloud.aio.bigquery call).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #66404:
URL: https://github.com/apache/airflow/pull/66404#issuecomment-4380858173

   Merged from `main` to resolve dependency on #66159, reviewing now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189901495


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -323,17 +326,34 @@ def serialize(self) -> tuple[str, dict[str, Any]]:
     async def run(self) -> AsyncIterator[TriggerEvent]:
         """Get current job execution status and yields a TriggerEvent with response data."""
         hook = self._get_async_hook()
+
         try:
             while True:
-                # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
+
                 if job_status["status"] == "success":
-                    query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
-                    records = hook.get_records(
-                        query_results=query_results,
-                        as_dict=self.as_dict,
-                        selected_fields=self.selected_fields,
-                    )
+                    sync_hook = await hook.get_sync_hook()
+
+                    if sync_hook.is_default_universe():
+                        query_results = await hook.get_job_output(
+                            job_id=self.job_id, project_id=self.project_id
+                        )
+                        records = hook.get_records(
+                            query_results=query_results,
+                            as_dict=self.as_dict,
+                            selected_fields=self.selected_fields,
+                        )
+                    else:
+                        query_results_args = {
+                            "job_id": self.job_id,
+                            "location": self.location,
+                            "selected_fields": self.selected_fields,
+                            "project_id": self.project_id,
+                        }
+                        records = await sync_to_async(sync_hook.get_query_results)(**query_results_args)

Review Comment:
   But this code will run only in Private Cloud, there  so the location should always be provided, otherwise it will not work anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adjust GCP BigQuery triggers for the private cloud [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66404:
URL: https://github.com/apache/airflow/pull/66404#discussion_r3189914439


##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Not entirely sure that I understand your point - tou said that gcloud-aio lib does not yet support non-default universes, but the logic which handles it (`if sync_hook.is_default_universe():` ) seems to be missing.
   How does it come together?
   



##########
providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py:
##########
@@ -249,7 +249,9 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
         try:
             while True:
                 # Poll for job execution status
-                job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)
+                job_status = await hook.get_job_status(
+                    job_id=self.job_id, project_id=self.project_id, location=self.location
+                )
                 if job_status["status"] == "success":
                     query_results = await hook.get_job_output(job_id=self.job_id, project_id=self.project_id)
                     records = hook.get_records(query_results)

Review Comment:
   Not entirely sure that I understand your point - you said that gcloud-aio lib does not yet support non-default universes, but the logic which handles it (`if sync_hook.is_default_universe():` ) seems to be missing.
   How does it come together?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org