You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "bingqin2 (via GitHub)" <gi...@apache.org> on 2026/09/11 13:53:39 UTC

[PR] Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator [airflow]

bingqin2 opened a new pull request, #72953:
URL: https://github.com/apache/airflow/pull/72953

   `GCSTimeSpanFileTransformOperator` accepts `download_num_attempts` and `upload_num_attempts` ("Number of attempts to try to download/upload a single file"), but since #62196 moved the transfers from `GCSHook.download()` / `GCSHook.upload()` to direct `blob.download_to_filename()` / `blob.upload_from_filename()` calls, both values are stored and never read. Every blob gets exactly one attempt regardless of the setting, so a transient `GoogleCloudError` either fails the task or, with `*_continue_on_fail`, silently drops that file.
   
   This restores the retry behaviour the operator had through the hook: a small `_run_with_attempts` helper retries `GoogleCloudError` up to the configured number of attempts with the hook's exponential back-off (`2**attempt` seconds), and both worker functions go through it. The default of one attempt keeps today's behaviour, and the `*_continue_on_fail` handling around the futures is unchanged.
   
   **Changes**
   
   - `providers/google/src/airflow/providers/google/cloud/operators/gcs.py`: `_run_with_attempts` helper; `_download` and `_upload` honor `download_num_attempts` and `upload_num_attempts`
   - `providers/google/tests/unit/google/cloud/operators/test_gcs.py`: `test_download_honors_download_num_attempts` and `test_upload_honors_upload_num_attempts` (retry, give-up and single-attempt cases; `time.sleep` is patched and its calls asserted), plus a shared `_setup_transform_process` helper
   
   **Testing**
   
   - `providers/google`: `tests/unit/google/cloud/operators/test_gcs.py` (`TestGCSTimeSpanFileTransformOperator`, 28 tests)
   - mypy on the changed module, prek hooks on the changed files
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Claude Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions). I reviewed and understand all changes; the tests were run locally as listed above.
   
   ---
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator [airflow]

Posted by "bingqin2 (via GitHub)" <gi...@apache.org>.
bingqin2 commented on code in PR #72953:
URL: https://github.com/apache/airflow/pull/72953#discussion_r4146760328


##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -996,7 +997,11 @@ def _download(blob_name: str):
                     )
                 destination_file.parent.mkdir(parents=True, exist_ok=True)
 
-                blob.download_to_filename(filename=str(destination_file))
+                self._run_with_attempts(
+                    lambda: blob.download_to_filename(filename=str(destination_file)),
+                    num_attempts=self.download_num_attempts,
+                    description=f"Download of gs://{self.source_bucket}/{blob_name}",
+                )

Review Comment:
   Done: `_download` is now decorated with `tenacity.retry`, with two differences from the snippet.
   
   - The stop uses the operator argument, `stop_after_attempt(self.download_num_attempts)`, which works because the function is defined inside `execute`.
   - It sets `reraise=True`. Without it, tenacity raises `RetryError` after the last attempt, and the `except GoogleCloudError` around `future.result()` (the `download_continue_on_fail` handling) would stop catching failures, even with the default of one attempt.
   
   Only `GoogleCloudError` is retried, so the path-containment `ValueError` still fails at once. `wait_exponential(multiplier=2, max=60)` keeps the 2, 4, 8 s waits of `GCSHook.download`, and `before_sleep_log` logs each retry through the task logger.
   



##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1071,8 +1076,10 @@ def _upload(upload_file: Path):
 
                 blob = bucket.blob(blob_name=upload_file_name, chunk_size=self.chunk_size)
 
-                blob.upload_from_filename(
-                    filename=str(upload_file),
+                self._run_with_attempts(
+                    lambda: blob.upload_from_filename(filename=str(upload_file)),
+                    num_attempts=self.upload_num_attempts,
+                    description=f"Upload of {upload_file_name} to gs://{self.destination_bucket}",
                 )
 
                 return upload_file_name

Review Comment:
   Done, the same decorator on `_upload`, driven by `upload_num_attempts`.
   



##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1101,6 +1108,28 @@ def _upload(upload_file: Path):
 
             return files_uploaded
 
+    def _run_with_attempts(

Review Comment:
   Removed, together with the `time` and `Callable` imports it needed.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #72953:
URL: https://github.com/apache/airflow/pull/72953


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator [airflow]

Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on code in PR #72953:
URL: https://github.com/apache/airflow/pull/72953#discussion_r4132332549


##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1071,8 +1076,10 @@ def _upload(upload_file: Path):
 
                 blob = bucket.blob(blob_name=upload_file_name, chunk_size=self.chunk_size)
 
-                blob.upload_from_filename(
-                    filename=str(upload_file),
+                self._run_with_attempts(
+                    lambda: blob.upload_from_filename(filename=str(upload_file)),
+                    num_attempts=self.upload_num_attempts,
+                    description=f"Upload of {upload_file_name} to gs://{self.destination_bucket}",
                 )
 
                 return upload_file_name

Review Comment:
   Same goes here, we can again use `tenacity.retry` decorator.



##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -996,7 +997,11 @@ def _download(blob_name: str):
                     )
                 destination_file.parent.mkdir(parents=True, exist_ok=True)
 
-                blob.download_to_filename(filename=str(destination_file))
+                self._run_with_attempts(
+                    lambda: blob.download_to_filename(filename=str(destination_file)),
+                    num_attempts=self.download_num_attempts,
+                    description=f"Download of gs://{self.source_bucket}/{blob_name}",
+                )

Review Comment:
   Since we already have a private function here; maybe we can use the `tenacity.retry` decorator to have a consistence across the provider. 
   ```python
   @tenacity.retry(
       stop=tenacity.stop_after_attempt(5),
       wait=tenacity.wait_exponential(multiplier=1, max=10),
       retry=tenacity.retry_if_exception_type(ValueError),
   )
   def _download(blob_name: str):
       ...
   ```
   
   > of course you may need to change the arguments of the decorator.



##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1101,6 +1108,28 @@ def _upload(upload_file: Path):
 
             return files_uploaded
 
+    def _run_with_attempts(

Review Comment:
   If we use the `tenacity.retry` decorator; we can totally avoid this function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Honor download_num_attempts and upload_num_attempts in GCSTimeSpanFileTransformOperator [airflow]

Posted by "bingqin2 (via GitHub)" <gi...@apache.org>.
bingqin2 commented on PR #72953:
URL: https://github.com/apache/airflow/pull/72953#issuecomment-5915139282

   Thanks for the review, @molcay. The helper is gone in favour of `tenacity.retry` (details in the threads), and the 28 time-span unit tests pass, including the retry, give-up and single-attempt cases for both downloads and uploads.
   
   On system tests: I don't have a GCP project, so I haven't run `example_gcs_transform_timespan.py`. That test covers the happy path, where the default of one attempt keeps today's behaviour; the retry path only runs on transient `GoogleCloudError`s, which the unit tests simulate. If your team can run it on the system-test dashboard, that would confirm the real transfer path.
   
   If you have a moment, #72954 in the same provider (the IdP extra parameters that never reach the token request) has been waiting for a first review since the 11th.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org