You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "karthi-keyan-n (via GitHub)" <gi...@apache.org> on 2025/02/07 15:45:09 UTC
[I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
karthi-keyan-n opened a new issue, #46563:
URL: https://github.com/apache/airflow/issues/46563
### Apache Airflow version
2.10.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
While creating workflow in Databricks using `DatabricksWorkflowTaskGroup`, I have a situation to pass some parameters to spark submit which are passed as jinja template.
Sample task creation code with DatabricksWorkflowTaskGroup,
```
databricks_workflow = DatabricksWorkflowTaskGroup(
group_id="test_wf",
databricks_conn_id=DATABRICKS_CONNECTION_ID,
add_suffix_on_collision=True,
job_clusters=job_clusters_spec,
)
with databricks_workflow:
task1 = build_task_1(dag)
task2 = build_task_2(dag)
task1 >> task2
```
From the above sample workflow creation code, `build_task_1` and `build_task_2` returns task which is my custom build operator on extending `DatabricksTaskBaseOperator`.
Sample code for custom build operator,
```
class CustomDatabricksWorkflowTaskOperator(DatabricksTaskBaseOperator):
"""Custom operator wrapping DatabricksTaskOperator"""
template_fields = (
"application_args",
"spark_conf",
"driver_jvm_props",
)
def __init__(
self,
*args,
class_name: str = "",
spark_conf: dict[str, str] | str = None,
application_args: List[str] | str | None = None,
driver_jvm_props: dict[str, str] | None = None,
application_conf_file: str = "test.conf",
libraries: List[dict[str, str]] = None,
**kwargs,
):
self.class_name = class_name
self.application_conf_file = application_conf_file
self.library = libraries if libraries is not None else _get_default_libraries()
self.application_args = application_args
super().__init__(*args, **kwargs)
self.driver_jvm_props = driver_jvm_props
self.spark_conf = spark_cone
def _get_task_base_json(self) -> dict[str, Any]:
return {
"run_if": self.databricks_trigger_rule,
'spark_jar_task': {
'main_class_name': self.class_name,
parameters": [
self.application_args,
"{{ get_test_template_value(ti,'test_key') }}" // I have hard-coded here for ref. In ideal case, this comes as part of self.application_args which has been added in template_fields
]
},
'libraries': self.library,
}
def execute(self, context):
super().execute(context)
```
From the above code sample, `{{ get_test_template_value(ti,'test_key') }}` is not getting resolved at the time of `launch` task from DatabricksWorkflowTaskGroup. But the same value got resolved at the time of task execution in airflow.
Sample json from `launch` task:
```
"tasks": [
{
"task_key": "task1",
"depends_on": [],
"run_if": "NONE_FAILED",
"spark_jar_task": {
"main_class_name": "task1.jar",
"parameters": [
"app.conf",
"{{ get_test_template_value(ti,'test_key') }}"
]
},
"libraries": [
{
"jar": "test.jar"
}
],
"job_cluster_key": "job_cluster"
},
```
Note: `get_test_template_value` function is added as a macro while DAG creation.
Though I have included the `application_args` as `template_fields` section in the CustomDatabricksWorkflowTaskOperator, it is unable to resolve the template.
### What you think should happen instead?
Expectation is to jinja template resolution should happen while the launch tasks trigger API requests to Databricks to create the workflow. But since the template resolution is happening lazily here i.e., at the time of task execution in airflow which is late in this case as the respective task is already created in Databricks at the time of `launch` and Databricks doesn't have to way to get the template resolved value.
### How to reproduce
1. Create a DAG with `DatabricksWorkflowTaskGroup`
2. Add couple of tasks
3. Create a custom Databricks Task operator by extending `DatabricksTaskBaseOperator`
4. Add one of the constructor parameter in `template_fields` section which needs to be tested
5. Override `_get_task_base_json` method to return a templated field/value as of the return dictionary
6. Trigger DAG
7. In launch task, you can notice from logs that the parameter property will pass the string as-is without resolving the actual value.
8. In the actual task execution, the actual value can be seen in the rendered template section.
### Operating System
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)", NAME="Debian GNU/Linux"
### Versions of Apache Airflow Providers
NA
### Deployment
Other Docker-based deployment
### Deployment details
We are building our own docker image with base image as `apache/airflow:2.10.4-python3.10` and use the same for deployment.
### Anything else?
Other approaches which I tried inside `_get_task_base_json` method of the custom operation which is responsible for returning the task JSON.
1. Tried to force the render by using `super().render_template(self.application_args, context)` and `super().render_template_fields(context)` which unfortunately was not possible as the context was not part of the `_get_task_base_json` method scope.
2. As this is an abstract method and I had to override, unable to pass context object to the step1.
Due to the above constraint, render templating as well as pulling from xcom also is not supported. (In other words, all the capabilities which we get from using context object is not possible).
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-2643286467
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4211773384
cc @moomindani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4322044262
I've created an initial draft PR for this. I need to do a bit more testing, but this should take care of the issue.
https://github.com/apache/airflow/pull/65869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4212011090
Thanks for the cc @eladkal. I took a look at this.
The reported behavior looks plausible based on the code. The root cause would be that child task operators inside `DatabricksWorkflowTaskGroup` never have their template fields rendered before being converted to the Databricks workflow JSON.
Here's what happens:
1. `_CreateDatabricksWorkflowOperator` (the "launch" task) stores references to child task operator instances in `tasks_to_convert`.
2. When the launch task executes, Airflow's task runner renders **only the launch operator's own `template_fields`** before calling `execute()`.
3. Inside `execute()`, `create_workflow_json()` calls `_convert_to_databricks_workflow_task()` → `_get_task_base_json()` on each child task — but **no one has called `render_template_fields()` on those child operators**, so any Jinja expressions (`{{ ds }}`, `{{ params.x }}`, custom macros, etc.) would be sent to Databricks as raw strings.
Additionally, even if rendering were triggered, some key fields are missing from `template_fields`:
- `DatabricksTaskOperator.template_fields` does not include `task_config`
- `DatabricksNotebookOperator.template_fields` does not include `notebook_path`, `source`, or `notebook_packages`
The workarounds shared by @cshaddox and @aldouslpzchr are consistent with this analysis — they work because `notebook_params` and `extra_job_params` are rendered as part of the **launch operator's own** `template_fields`, bypassing the unrendered child tasks.
**To fix this**, `create_workflow_json()` (or `_convert_to_databricks_workflow_task()`) should call `render_template_fields(context)` on each child task before extracting the JSON, and the missing fields above should be added to `template_fields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4322016177
@moomindani, why do you think this?
> `DatabricksNotebookOperator.template_fields` does not include `notebook_path`, `source`, or `notebook_packages`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "aldouslpzchr (via GitHub)" <gi...@apache.org>.
aldouslpzchr commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-3248818659
I got around this issue by adding `extra_job_params` to template fields of `_CreateDatabricksWorkflowOperator` and passing them using workflow parameters
in plugins file:
```python
from airflow.providers.databricks.operators import databricks_workflow
class Patched_CreateDatabricksWorkflowOperator(_CreateDatabricksWorkflowOperator):
template_fields = (
"notebook_params",
"extra_job_params",
)
databricks_workflow._CreateDatabricksWorkflowOperator = (
Patched_CreateDatabricksWorkflowOperator
)
class DatabricksProviderPatcherPlugin(AirflowPlugin):
name = "databricks_provider_patcher_plugin"
```
`DatabricksWorkflowTaskGroup`
```python
...
with DatabricksWorkflowTaskGroup(
databricks_conn_id="dataricks_default",
group_id="workflow",
extra_job_params={
"parameters": [
{
"name": "date",
"default": "{{ tomorrow_ds }}",
},
],
},
) as workflow_tg:
...
```
`DatabricksTaskOperator`
```python
task = DatabricksTaskOperator(
task_id="task",
task_config={
"spark_python_task": {
"python_file": "...",
"parameters": [
"--date",
"{{ job.parameters.date }}",
],
},
},
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4322156338
PR is updated, should be ready to go (pending passage of checks)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4321989706
I'm taking a quick peek at this as well, but just want to call out the following.
> `DatabricksTaskOperator.template_fields` does not include `task_config`.
This has been taken care of by #65858.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4346009974
No worries - I'll go ahead and open up another PR for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "gabrielbckrsc (via GitHub)" <gi...@apache.org>.
gabrielbckrsc commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-3458961628
Having the same issue using other jinja templates default to airflow
`{{ ds }}`
and
`{{ param.date }}`
```python
data_preparation = DatabricksTaskOperator(
dag=dag,
task_id=...
databricks_conn_id='databricks_conn',
task_config=dict(
python_wheel_task=dict(
package_name=...
entry_point=...
parameters=[
'--date-range-min', '{{ ds }}',
]
)
)
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "cshaddox (via GitHub)" <gi...@apache.org>.
cshaddox commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-2960543382
Experienced the same behaviors, very annoying. I got around it with a hacky fix that uses notebook_params, which do get templated properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Jinja templating capability is not working for the parameter (additional args) in spark job submit for DatabricksWorkflowTaskGroup [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on issue #46563:
URL: https://github.com/apache/airflow/issues/46563#issuecomment-4332715234
@jroachgolf84 Good catch on #65858, thanks!
On `DatabricksNotebookOperator` — looking at it again, the strongest case is `notebook_path` (users may want to use Jinja expressions like `{{ ds }}` in the path). `source` and `notebook_packages` are less likely to need templating, so I overstated that part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org