You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tnk-ysk (via GitHub)" <gi...@apache.org> on 2025/01/17 15:41:49 UTC
[PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
tnk-ysk opened a new pull request, #45754:
URL: https://github.com/apache/airflow/pull/45754
Add deferrable mode to google cloud storage transfer sensor and operators.
- CloudDataTransferServiceS3ToGCSOperator
- CloudDataTransferServiceGCSToGCSOperator
- CloudDataTransferServiceJobStatusSensor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45754:
URL: https://github.com/apache/airflow/pull/45754#discussion_r1938265175
##########
providers/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py:
##########
@@ -596,3 +598,112 @@ async def get_latest_operation(self, job: TransferJob) -> Message | None:
operation = TransferOperation.deserialize(response_operation.metadata.value)
return operation
return None
+
+ async def list_transfer_operations(
+ self,
+ request_filter: dict | None = None,
+ **kwargs,
+ ) -> list[TransferOperation]:
+ """
+ Get a transfer operation in Google Storage Transfer Service.
+
+ :param request_filter: (Required) A request filter, as described in
+ https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
+ With one additional improvement:
+ :return: transfer operation
+
+ The ``project_id`` parameter is optional if you have a project ID
+ defined in the connection. See: :doc:`/connections/gcp`
+ """
+ # To preserve backward compatibility
+ # TODO: remove one day
+ if request_filter is None:
+ if "filter" in kwargs:
+ request_filter = kwargs["filter"]
+ if not isinstance(request_filter, dict):
+ raise ValueError(f"The request_filter should be dict and is {type(request_filter)}")
+ warnings.warn(
+ "Use 'request_filter' instead of 'filter'",
+ AirflowProviderDeprecationWarning,
Review Comment:
We can always make a follow-up change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #45754:
URL: https://github.com/apache/airflow/pull/45754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
Posted by "tnk-ysk (via GitHub)" <gi...@apache.org>.
tnk-ysk commented on code in PR #45754:
URL: https://github.com/apache/airflow/pull/45754#discussion_r1922424964
##########
providers/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py:
##########
@@ -596,3 +598,112 @@ async def get_latest_operation(self, job: TransferJob) -> Message | None:
operation = TransferOperation.deserialize(response_operation.metadata.value)
return operation
return None
+
+ async def list_transfer_operations(
+ self,
+ request_filter: dict | None = None,
+ **kwargs,
+ ) -> list[TransferOperation]:
+ """
+ Get a transfer operation in Google Storage Transfer Service.
+
+ :param request_filter: (Required) A request filter, as described in
+ https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
+ With one additional improvement:
+ :return: transfer operation
+
+ The ``project_id`` parameter is optional if you have a project ID
+ defined in the connection. See: :doc:`/connections/gcp`
+ """
+ # To preserve backward compatibility
+ # TODO: remove one day
+ if request_filter is None:
+ if "filter" in kwargs:
+ request_filter = kwargs["filter"]
+ if not isinstance(request_filter, dict):
+ raise ValueError(f"The request_filter should be dict and is {type(request_filter)}")
+ warnings.warn(
+ "Use 'request_filter' instead of 'filter'",
+ AirflowProviderDeprecationWarning,
Review Comment:
It's not that I want to add deprecated code, but to make it compatible with non-deferrable mode.
https://github.com/apache/airflow/blob/25aeb1166e6e8d093892ad3a7b1a341375b0cf51/providers/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py#L255-L267
I'll delete it if it is unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #45754:
URL: https://github.com/apache/airflow/pull/45754#issuecomment-2602003678
Hi! Thank you for preparing PR with changes :)
Can you please add a system tests for this? As a new task with deferrable mode for every operator that you have modified.
Can you please provide also screenshots of successfully executed system tests with your changes? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to google cloud storage transfer sensor and operators [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45754:
URL: https://github.com/apache/airflow/pull/45754#discussion_r1938265128
##########
providers/src/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py:
##########
@@ -596,3 +598,112 @@ async def get_latest_operation(self, job: TransferJob) -> Message | None:
operation = TransferOperation.deserialize(response_operation.metadata.value)
return operation
return None
+
+ async def list_transfer_operations(
+ self,
+ request_filter: dict | None = None,
+ **kwargs,
+ ) -> list[TransferOperation]:
+ """
+ Get a transfer operation in Google Storage Transfer Service.
+
+ :param request_filter: (Required) A request filter, as described in
+ https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
+ With one additional improvement:
+ :return: transfer operation
+
+ The ``project_id`` parameter is optional if you have a project ID
+ defined in the connection. See: :doc:`/connections/gcp`
+ """
+ # To preserve backward compatibility
+ # TODO: remove one day
+ if request_filter is None:
+ if "filter" in kwargs:
+ request_filter = kwargs["filter"]
+ if not isinstance(request_filter, dict):
+ raise ValueError(f"The request_filter should be dict and is {type(request_filter)}")
+ warnings.warn(
+ "Use 'request_filter' instead of 'filter'",
+ AirflowProviderDeprecationWarning,
Review Comment:
I am merging it as is now - before we migrate Google Provider to the new structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org