You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "osintalex (via GitHub)" <gi...@apache.org> on 2026/07/29 11:52:08 UTC
[PR] ...fix: update template fields [airflow]
osintalex opened a new pull request, #70679:
URL: https://github.com/apache/airflow/pull/70679
Allow callers to template both gcs, bq inputs and outputs for batch jobs. I am adding this in after swapping from using the vertex batch ai job operator to writing my own custom operator due to the inability to template destinations and outputs to both gcs and bq.
This seems like normal airflow behaviour to me to allow templating for different inputs/outputs, i.e. in dev environment you would inject a dev bigquery project from the config and then template that to build `<dev-project>.<dataset>` whereas in prod you would have `<prod-project>.<dataset>`. Lacking the ability to do this in my experience pretty much breaks this operator for multi-stage work environments where you want the same code to run across dev/staging/prod.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
No
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow templating GCS and BigQuery URIs in Vertex AI batch prediction [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #70679:
URL: https://github.com/apache/airflow/pull/70679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add more template fields to Create Batch Vertex AI Job Operator to support bq, gcs templating [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #70679:
URL: https://github.com/apache/airflow/pull/70679#discussion_r3995729626
##########
providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/batch_prediction_job.py:
##########
@@ -159,7 +159,18 @@ class CreateBatchPredictionJobOperator(GoogleCloudBaseOperator):
:param poll_interval: Interval size which defines how often job status is checked in deferrable mode.
"""
- template_fields = ("region", "project_id", "model_name", "impersonation_chain", "job_display_name")
+ template_fields = (
+ "region",
+ "project_id",
+ "model_name",
+ "impersonation_chain",
+ "job_display_name",
+ "gcs_source",
+ "bigquery_source",
+ "gcs_destination_prefix",
+ "gcs_destination_prefix",
Review Comment:
Apologies for the delay, and thanks for the detailed answer!
I'm satisfied with your current explanation. The implications are mainly regaulations around what you can do with template fields in the constructor - once a field is templated, you cannot make any validations for its value in the operator's constructor.
In this case it should be safe, so it's a LGTM from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow templating GCS and BigQuery URIs in Vertex AI batch prediction [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #70679:
URL: https://github.com/apache/airflow/pull/70679#issuecomment-5644916711
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add more template fields to Create Batch Vertex AI Job Operator to support bq, gcs templating [airflow]
Posted by "osintalex (via GitHub)" <gi...@apache.org>.
osintalex commented on code in PR #70679:
URL: https://github.com/apache/airflow/pull/70679#discussion_r3681878916
##########
providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/batch_prediction_job.py:
##########
@@ -159,7 +159,18 @@ class CreateBatchPredictionJobOperator(GoogleCloudBaseOperator):
:param poll_interval: Interval size which defines how often job status is checked in deferrable mode.
"""
- template_fields = ("region", "project_id", "model_name", "impersonation_chain", "job_display_name")
+ template_fields = (
+ "region",
+ "project_id",
+ "model_name",
+ "impersonation_chain",
+ "job_display_name",
+ "gcs_source",
+ "bigquery_source",
+ "gcs_destination_prefix",
+ "gcs_destination_prefix",
Review Comment:
Ty will catch.
Regarding this https://github.com/apache/airflow/issues/70296
I don't think I really understand the implications of this. From what I can tell in the operator the `__init__` method simply assigns these attributes onto the class https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/batch_prediction_job.py#L165.
I'm struggling to see how that would cause some sort of issue or antipattern here, it doesn't look like any anything happens in the constructor here apart from the assignment, which I believe is in alignment with your guide here https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html#limitations. Let me know if I've misunderstood though.
As for 2, yes I ran into this in production when running a batch inference job with sources in bigquery and destination in bigquery. It made in my experience the operator hard to understand and use since the bigquery source and destinations were not templatable, which I found counterintuitive. By comparison these sorts of fields are typically templatable across airflow, e.g. on bigquery insert job operators. It was also very difficult to debug since the job just mysteriously fails and you don't get any error feedback on the source/destination being invalid.
This is what I was trying to articulate by 'normal', i.e. allowing these fields to be templatable in my opinion is much more consistent with most other airflow operators, where source/destination outputs to GCS or BQ are typically templatable.
Lacking the ability to template the operator makes it, in my opinion, quite unusable in most professional production environments where you would have dev/staging/prod environments that all need different input/output configurations but should run the same code.
While I could create a new class that inherits from it, that doesn't feel so good to me since I would argue most people using this operator in production systems will want the same thing as me, i.e. ability to template these fields dynamically based on environment. Seems to me like something which should be supported by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ...fix: update template fields [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #70679:
URL: https://github.com/apache/airflow/pull/70679#discussion_r3674186462
##########
providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/batch_prediction_job.py:
##########
@@ -159,7 +159,18 @@ class CreateBatchPredictionJobOperator(GoogleCloudBaseOperator):
:param poll_interval: Interval size which defines how often job status is checked in deferrable mode.
"""
- template_fields = ("region", "project_id", "model_name", "impersonation_chain", "job_display_name")
+ template_fields = (
+ "region",
+ "project_id",
+ "model_name",
+ "impersonation_chain",
+ "job_display_name",
+ "gcs_source",
+ "bigquery_source",
+ "gcs_destination_prefix",
+ "gcs_destination_prefix",
Review Comment:
This appears twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org