You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yassun7010 (via GitHub)" <gi...@apache.org> on 2024/02/07 09:21:28 UTC
[I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
yassun7010 opened a new issue, #37217:
URL: https://github.com/apache/airflow/issues/37217
### Description
[CloudBatchSubmitJobOperator](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/cloud_batch.html)'s `runnable.container.commands` is not expanded when the jinja template is used as a command argument.
### Use case/motivation
I would like to pass `logical_date` to a script args to run in CloudBatch.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-3267825676
> In case somebody encounters this issue, a quick & dirty hack is to extend the `CloudBatchSubmitJobOperator` class by creating a custom class that inherits from it.
@raph-p it is not hard to open a PR, if it is a single line to change you can even make this via Github UI. Also my OSS Contribution started like this :-D
You can make it also as describe above with a single line of code. From the instantiated operator in your Dag you can also directly adjust the `template_fields` attribute and extend the filed in one line of code in Dag code, no need to inherit a custom class from the original one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "yassun7010 (via GitHub)" <gi...@apache.org>.
yassun7010 commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2060652172
@romsharon98
Please close, Thank you!
After some trial and error, we have switched to an implementation using task flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "yassun7010 (via GitHub)" <gi...@apache.org>.
yassun7010 commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-1933186780
I am looking at this [PR](https://github.com/apache/airflow/pull/20905).
I had added `job` to `template_fields` but it did not work.
Cloud Batch job is a deep configuration class and I need to template the contents of `self.job.task_groups[0].task_spec.runnables[0].container.commands`.
I have assigned execution arguments to `self.commands` and passed them to the `self.job` creation process (hoping they would be shared as a list), but this method is not working either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "raph-p (via GitHub)" <gi...@apache.org>.
raph-p commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-3266776436
In case somebody encountered this issue, a quick & dirty hack is to extend the `CloudBatchSubmitJobOperator` class by creating a custom class that inherits from it. It adds the whole `job` field as template, allowing to pass the command (or env vars, or even more) as jinja template.
We also decided to add the `job_name`as well as the job_name, in case some would pass a dynamic name (i.e. including the execution date for example), as the name of a Batch Job must be unique.
I'm not sure I'll have the time to do a fix in the `airflow.provider` myself, as I've never contributed to an open-source project and I'm not sure where to start.
```python
class CustomCloudBatchSubmitJobOperator(CloudBatchSubmitJobOperator):
"""
Extends CloudBatchSubmitJobOperator by allowing jinja substitution of the "job_name" and "job"
"""
template_fields = ("project_id", "region", "gcp_conn_id", "impersonation_chain", "job_name", "job"). # this is the important line
def __init__(
self,
project_id: str,
region: str,
job_name: str,
job: dict | Job,
polling_period_seconds: float = 10,
timeout_seconds: float | None = None,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: str | Sequence[str] | None = None,
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
**kwargs,
) -> None:
super().__init__(
project_id=project_id,
region=region,
job_name=job_name,
job=job,
polling_period_seconds=polling_period_seconds,
timeout_seconds=timeout_seconds,
gcp_conn_id=gcp_conn_id,
impersonation_chain=impersonation_chain,
deferrable=deferrable,
**kwargs,
)
```
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2060541478
> Thanks for letting me know!
>
>
>
> Next week I will try to work on it.
Can we close this issue? @yassun7010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-1937108873
Yes, (I am not an expert in this operator but...) if you take look to the docs, the Operator accepts either a `dict` or a class as Job definition. If you pass a dict then I assume the dict will implicitly be parsed to the object structure describing the workload/Job.
Means in the instance of the operator change the `template_fields` variable and add the `job` field, pass the needed dict as string with the template fields. The template engine will parse the dict-string as template, replace the field. You need to define `render_template_as_native_obj=True` in the DAG as described in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html#referencing-params-in-a-task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "raph-p (via GitHub)" <gi...@apache.org>.
raph-p commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2518249018
Hi team,
Sorry to reopen this issue. We actually encountered the exact same issue and after an afternoon digging the operator code, we came across this issue.
We have the same setup, where we want to pass a jinja template to the CloudBatchSubmitJobOperator (to pass the current date). At first we tried passing a job object as mentioned in the airflow/google provider docs, and then we tested passing the job as a dict as suggested by @jscheffl
**Passing as a dict produces** the same result as the job object, ie. **the template is not expanded/substituted**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-1931612865
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 closed issue #37217: CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates
URL: https://github.com/apache/airflow/issues/37217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2518576724
@yassun7010 can you share details helping @raph-p to resolve the problem as well? Is there a clarification in documentation missing? Then we would be happy to receive a documentation contribution, e.g. via https://github.com/apache/airflow/edit/main/docs/apache-airflow/core-concepts/params.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #37217: CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates
URL: https://github.com/apache/airflow/issues/37217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-1932988640
Reason for not templating is probably that the Google Operator in https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/operators/cloud_batch.py#L60 does not describe the "command" field as template field (see https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html#templating)
From the descriptio of this field it would make sense to have this template-able. If you have a strong demand, please contribute a PR to the respective code section, else maybe you need to wait that somebody picks this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "yassun7010 (via GitHub)" <gi...@apache.org>.
yassun7010 commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-1937397021
Thanks for letting me know!
Next week I will try to work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "yassun7010 (via GitHub)" <gi...@apache.org>.
yassun7010 commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2520237004
My solution is as follows:
## common code
```python
from typing import TYPE_CHECKING, cast
from airflow.decorators import task
from airflow import XComArg
if TYPE_CHECKING:
import google.cloud.batch_v1 as batch_v1
@task
def submit_cloud_batch_job(
job_id: str,
commands: list[str] | XComArg,
/,
*,
envs: dict[str, str] | None = None,
) -> str:
client = batch_v1.BatchServiceClient()
create_request = batch_v1.CreateJobRequest()
runnable = batch_v1.Runnable()
...
runnable.container.commands = cast(list[str], commands)
...
client.create_job(request=create_request)
return create_request.job_id
```
## DAG code
```python
import datetime
from pathlib import Path
from airflow.decorators import dag, task
from airflow.models import DagRun
from common_code import submit_cloud_batch_job
@dag(
dag_id=Path(__file__).stem,
schedule_interval="0 0 * * *",
start_date=datetime.datetime(2021, 1, 1),
)
def execute():
@task()
def make_commands(dag_run: DagRun | None = None):
conf = dag_run.conf if dag_run else {}
commands = [
"python",
"-m",
"my-task",
]
# You can pass any command argument you like from conf or Python code to Cloud Batch.
if value := conf.get("my-option"):
commands.extend(["--my-option", str(value)])
return commands
submit_cloud_batch_job(
Path(__file__).stem,
make_commands(),
)
execute()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] CloudBatchSubmitJobOperator runnable.container.commands does not interpret jinja templates [airflow]
Posted by "raph-p (via GitHub)" <gi...@apache.org>.
raph-p commented on issue #37217:
URL: https://github.com/apache/airflow/issues/37217#issuecomment-2520247562
Thank you so much @yassun7010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org