You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "olegkachur-e (via GitHub)" <gi...@apache.org> on 2026/05/14 08:27:01 UTC
[PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
olegkachur-e opened a new pull request, #66917:
URL: https://github.com/apache/airflow/pull/66917
- moidfy trigger
- slight hooks refactoring
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66917:
URL: https://github.com/apache/airflow/pull/66917#discussion_r3246998333
##########
providers/google/tests/unit/google/cloud/triggers/test_cloud_sql.py:
##########
@@ -48,8 +48,18 @@ def trigger():
)
+@pytest.fixture
+def sync_hook_mock():
+ mock_obj = mock.MagicMock()
Review Comment:
Sure, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66917:
URL: https://github.com/apache/airflow/pull/66917#discussion_r3246997030
##########
providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py:
##########
@@ -41,13 +43,15 @@ def __init__(
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: str | Sequence[str] | None = None,
poke_interval: int = 20,
+ api_version: str = "v1beta4",
):
super().__init__()
self.gcp_conn_id = gcp_conn_id
self.impersonation_chain = impersonation_chain
self.operation_name = operation_name
self.project_id = project_id
self.poke_interval = poke_interval
+ self.api_version = api_version
Review Comment:
Thank you for the common solution, I've updated trigger and test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #66917:
URL: https://github.com/apache/airflow/pull/66917#discussion_r3243169032
##########
providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py:
##########
@@ -41,13 +43,15 @@ def __init__(
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: str | Sequence[str] | None = None,
poke_interval: int = 20,
+ api_version: str = "v1beta4",
):
super().__init__()
self.gcp_conn_id = gcp_conn_id
self.impersonation_chain = impersonation_chain
self.operation_name = operation_name
self.project_id = project_id
self.poke_interval = poke_interval
+ self.api_version = api_version
Review Comment:
Triggers must include all constructor arguments in their `serialize()` return value.
Please fix it here, I'll work on a generic pre-commit to fix existing bugged triggers and prevent that for the next time.
##########
providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py:
##########
@@ -432,13 +441,8 @@ def _wait_for_operation_to_complete(
:param time_to_sleep: Time to sleep between active checks of the operation results.
:return: None
"""
- service = self.get_conn()
Review Comment:
nit: `self.get_conn()` will be called in every iteration of the `while` loop. It is now cached internally, but to prevent potential regressions I'd keep the original and pass it as parameter where needed.
##########
providers/google/tests/unit/google/cloud/triggers/test_cloud_sql.py:
##########
@@ -148,3 +161,28 @@ async def test_async_export_trigger_exception_should_execute_successfully(
generator = trigger.run()
actual = await generator.asend(None)
assert TriggerEvent({"status": "failed", "message": "Test exception"}) == actual
+
+ @pytest.mark.asyncio
+ @mock.patch(HOOK_STR.format("CloudSQLAsyncHook.get_operation"))
+ async def test_async_export_trigger_executest_successfully_in_custom_universe(
Review Comment:
```suggestion
async def test_async_export_trigger_executes_successfully_in_custom_universe(
```
##########
providers/google/tests/unit/google/cloud/triggers/test_cloud_sql.py:
##########
@@ -48,8 +48,18 @@ def trigger():
)
+@pytest.fixture
+def sync_hook_mock():
+ mock_obj = mock.MagicMock()
Review Comment:
nit: can it be mocked with a spec definition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #66917:
URL: https://github.com/apache/airflow/pull/66917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on code in PR #66917:
URL: https://github.com/apache/airflow/pull/66917#discussion_r3246920368
##########
providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py:
##########
@@ -432,13 +441,8 @@ def _wait_for_operation_to_complete(
:param time_to_sleep: Time to sleep between active checks of the operation results.
:return: None
"""
- service = self.get_conn()
Review Comment:
I'm not against the idea of fighting regression but it makes this method usage really weird in other places, e.g. in trigger or in the future, since every other method calls `self.get_conn()` internally.
I'll add the test to ensure `get_conn` being cached, I think it will help with regression, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adapt GCP CloudSQL trigger to run in private cloud [airflow]
Posted by "olegkachur-e (via GitHub)" <gi...@apache.org>.
olegkachur-e commented on PR #66917:
URL: https://github.com/apache/airflow/pull/66917#issuecomment-4458393176
@shahar1 Thank you for the detailed review and serialization catch, I believe I've addressed the all comments, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org