You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mitre88 (via GitHub)" <gi...@apache.org> on 2026/07/27 00:24:08 UTC
[PR] Apply GCS operator template-field logic after rendering [airflow]
mitre88 opened a new pull request, #70488:
URL: https://github.com/apache/airflow/pull/70488
Three operators in `google/cloud/operators/gcs.py` applied logic to template fields in `__init__`, before Jinja rendering:
- `GCSListObjectsOperator` warned about the deprecated `delimiter` at Dag parse time — a templated `delimiter` that renders empty warned anyway. The warning now fires in `execute()` based on the rendered value.
- `GCSDeleteObjectsOperator` enforced the `objects`/`prefix` mutual exclusion at construction time, rejecting templated combinations before rendering. The check now runs at the top of `execute()` (also normalizing the `super().__init__` call to run first).
- `GCSFileTransformOperator` resolved `destination_bucket`/`destination_object` fallbacks to source in `__init__`, so a templated source was copied un-rendered into the destination. The fallback now resolves in `execute()` (and in `get_openlineage_facets_on_start`, which can run before execute).
All three classes are removed from the exemption list.
Note: `TestGCSDeleteObjectsOperator::test_get_openlineage_facets_on_start[objects]` fails identically on current `main` in my environment (pre-existing, unrelated).
related: #70296
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5)
Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve GCSFileTransformOperator destination fallbacks after rendering [airflow]
Posted by "mitre88 (via GitHub)" <gi...@apache.org>.
mitre88 commented on PR #70488:
URL: https://github.com/apache/airflow/pull/70488#issuecomment-5147959429
@shahar1 Done — the PR is now scoped to `GCSFileTransformOperator` only and rebased on current `main` (which includes #70505).
To confirm the remaining change is consistent with the #70505 distinction: `destination_bucket or self.source_bucket` is a truthiness/value read of the unrendered template parameter, **not** an argument-provision check (`is None`/`is not None`), so it is not covered by the new sanctioned pattern — the exemption entry only stays valid if the fallback moves out of `__init__`. That's what this PR does:
- `__init__` now stores `destination_bucket`/`destination_object` verbatim (no read of the unrendered value).
- The `or source_*` fallback resolves in `execute()`, after rendering.
- `get_openlineage_facets_on_complete` (which can run outside `execute`) resolves the fallbacks locally.
- The `GCSFileTransformOperator` exemption entry is removed; `validate_operators_init.py` passes on the file (exit 0).
Everything else from the earlier revision (`GCSDeleteObjectsOperator`, `GCSListObjectsOperator`, their exemption entries) was dropped.
---
Drafted-by: Claude Code (Fable 5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply GCS operator template-field logic after rendering [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #70488:
URL: https://github.com/apache/airflow/pull/70488#discussion_r3655775955
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -311,25 +311,20 @@ def __init__(
impersonation_chain: str | Sequence[str] | None = None,
**kwargs,
) -> None:
+ super().__init__(**kwargs)
self.bucket_name = bucket_name
self.objects = objects
self.prefix = prefix
self.ignore_error = ignore_error
self.gcp_conn_id = gcp_conn_id
self.impersonation_chain = impersonation_chain
- if objects is None and prefix is None:
- err_message = "(Task {task_id}) Either objects or prefix should be set. Both are None.".format(
- **kwargs
- )
- raise ValueError(err_message)
- if objects is not None and prefix is not None:
- err_message = "(Task {task_id}) Objects or prefix should be set. Both provided.".format(**kwargs)
- raise ValueError(err_message)
-
- super().__init__(**kwargs)
-
Review Comment:
Please restore the changes here, as it will be flagged as false-positive (see #https://github.com/apache/airflow/pull/70505 for more details)
##########
scripts/ci/prek/validate_operators_init_exemptions.txt:
##########
@@ -25,8 +25,6 @@ providers/google/src/airflow/providers/google/cloud/operators/cloud_storage_tran
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocCreateClusterOperator
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocSubmitJobOperator
providers/google/src/airflow/providers/google/cloud/operators/functions.py::CloudFunctionDeployFunctionOperator
-providers/google/src/airflow/providers/google/cloud/operators/gcs.py::GCSDeleteObjectsOperator
Review Comment:
Please restore this lien
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -311,25 +311,20 @@ def __init__(
impersonation_chain: str | Sequence[str] | None = None,
**kwargs,
) -> None:
+ super().__init__(**kwargs)
self.bucket_name = bucket_name
self.objects = objects
self.prefix = prefix
self.ignore_error = ignore_error
self.gcp_conn_id = gcp_conn_id
self.impersonation_chain = impersonation_chain
- if objects is None and prefix is None:
- err_message = "(Task {task_id}) Either objects or prefix should be set. Both are None.".format(
- **kwargs
- )
- raise ValueError(err_message)
- if objects is not None and prefix is not None:
- err_message = "(Task {task_id}) Objects or prefix should be set. Both provided.".format(**kwargs)
- raise ValueError(err_message)
-
- super().__init__(**kwargs)
-
def execute(self, context: Context) -> None:
+ if self.objects is None and self.prefix is None:
+ raise ValueError(f"(Task {self.task_id}) Either objects or prefix should be set. Both are None.")
+ if self.objects is not None and self.prefix is not None:
+ raise ValueError(f"(Task {self.task_id}) Objects or prefix should be set. Both provided.")
+
Review Comment:
Please revert these changes
##########
scripts/ci/prek/validate_operators_init_exemptions.txt:
##########
@@ -25,8 +25,6 @@ providers/google/src/airflow/providers/google/cloud/operators/cloud_storage_tran
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocCreateClusterOperator
providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocSubmitJobOperator
providers/google/src/airflow/providers/google/cloud/operators/functions.py::CloudFunctionDeployFunctionOperator
-providers/google/src/airflow/providers/google/cloud/operators/gcs.py::GCSDeleteObjectsOperator
Review Comment:
Please restore this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve GCSFileTransformOperator destination fallbacks after rendering [airflow]
Posted by "mitre88 (via GitHub)" <gi...@apache.org>.
mitre88 commented on PR #70488:
URL: https://github.com/apache/airflow/pull/70488#issuecomment-5091953627
Done — reverted the `GCSDeleteObjectsOperator` changes (code, test, and exemption entry restored to match main) per the relaxed provision-check definition in #70505. The PR now only moves the `GCSFileTransformOperator` destination fallbacks to execute time; title and description updated accordingly.
---
Drafted-by: Claude Code (Fable 5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply GCS operator template-field logic after rendering [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #70488:
URL: https://github.com/apache/airflow/pull/70488#discussion_r3653943544
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -229,18 +229,18 @@ def __init__(
super().__init__(**kwargs)
self.bucket = bucket
self.prefix = prefix
- if delimiter:
- warnings.warn(
- "Usage of 'delimiter' is deprecated, please use 'match_glob' instead. Planned removal date: October 5, 2026.",
- AirflowProviderDeprecationWarning,
- stacklevel=2,
- )
Review Comment:
I think this should still be in `__init__`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply GCS operator template-field logic after rendering [airflow]
Posted by "mitre88 (via GitHub)" <gi...@apache.org>.
mitre88 commented on code in PR #70488:
URL: https://github.com/apache/airflow/pull/70488#discussion_r3654663675
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -229,18 +229,18 @@ def __init__(
super().__init__(**kwargs)
self.bucket = bucket
self.prefix = prefix
- if delimiter:
- warnings.warn(
- "Usage of 'delimiter' is deprecated, please use 'match_glob' instead. Planned removal date: October 5, 2026.",
- AirflowProviderDeprecationWarning,
- stacklevel=2,
- )
Review Comment:
Agreed — the deprecation applies to any usage of the argument, not the rendered value. Moved the warning back to `__init__` as it is on main, restored the `GCSListObjectsOperator` exemption entry, and dropped the execute-time warning test. The other two changes (objects/prefix validation in `GCSDeleteObjectsOperator` and destination fallbacks in `GCSFileTransformOperator`) are unchanged.
---
Drafted-by: Claude Code (Fable 5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve GCSFileTransformOperator destination fallbacks after rendering [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #70488:
URL: https://github.com/apache/airflow/pull/70488#issuecomment-5751592468
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/35527536641'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/e75094dbafcaa653fc7934a99b2a9be4f027b7ef"><img src='https://img.shields.io/badge/Commit-e75094d-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker e75094d v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve GCSFileTransformOperator destination fallbacks after rendering [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #70488:
URL: https://github.com/apache/airflow/pull/70488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org