You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dheerajkumar-solanki (via GitHub)" <gi...@apache.org> on 2025/03/20 02:48:49 UTC
[I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
dheerajkumar-solanki opened a new issue, #47983:
URL: https://github.com/apache/airflow/issues/47983
### Apache Airflow Provider(s)
databricks
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.24.0
apache-airflow-providers-databricks==7.2.1
databricks-sql-connector==4.0.0
### Apache Airflow version
2.10.3
### Operating System
MacOs
### Deployment
Docker-Compose
### Deployment details
Testing using [AWS MWAA Local Runner](https://github.com/aws/aws-mwaa-local-runner)
### What happened
When upgraded the databricks provider to version 7.2.1 and tested the DAG with DatabricksNotebookOperator, DAG is failing with the below error:
```airflow.exceptions.AirflowException: Response: {"error_code":"INVALID_PARAMETER_VALUE","message":"Invalid dependency graph, task 'e771895875324e2902a93fdc2ff36326' can not reference itself."}, Status Code: 400```
<img width="1501" alt="Image" src="https://github.com/user-attachments/assets/e90a1526-f426-448f-a547-8319d2fa7369" />
### What you think should happen instead
Ideally, It should generate correct dependency between Databricks tasks and deploy the Databricks Job. Whereas it is generating the wrong dependency graph for databricks tasks (referencing the dependent task itself).
### How to reproduce
Below is the DAG code to regenerate the error:
```
import os
from airflow.models.dag import DAG
from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator
from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup
from airflow.utils.timezone import datetime
DATABRICKS_CONN_ID = os.getenv("DATABRICKS_CONN_ID", "one_databricks")
job_cluster_spec = [
{
"job_cluster_key": "Shared_job_cluster",
"new_cluster": {
"cluster_name": "",
"spark_version": "11.3.x-scala2.12",
"num_workers": 1,
"spark_conf": {},
"node_type_id": "r3.xlarge",
"ssh_public_keys": [],
"custom_tags": {},
"spark_env_vars": {"PYSPARK_PYTHON": "/databricks/python3/bin/python3"},
"cluster_source": "JOB",
"init_scripts": [],
},
}
]
dag = DAG(
dag_id="example_databricks_workflow",
start_date=datetime(2022, 1, 1),
schedule=None,
catchup=False,
)
with dag:
task_group = DatabricksWorkflowTaskGroup(
group_id=f"test_workflow",
databricks_conn_id=DATABRICKS_CONN_ID,
job_clusters=job_cluster_spec,
)
with task_group:
notebook_1 = DatabricksNotebookOperator(
task_id="workflow_notebook_1",
databricks_conn_id=DATABRICKS_CONN_ID,
notebook_path="/Shared/Notebook_1",
source="WORKSPACE",
job_cluster_key="Shared_job_cluster",
)
notebook_2 = DatabricksNotebookOperator(
task_id="workflow_notebook_2",
databricks_conn_id=DATABRICKS_CONN_ID,
notebook_path="/Shared/Notebook_2",
source="WORKSPACE",
job_cluster_key="Shared_job_cluster",
)
notebook_1 >> notebook_2
```
Code is from Databricks example dags: https://github.com/apache/airflow/blob/providers-databricks/7.2.1/providers/databricks/tests/system/databricks/example_databricks_workflow.py
### Anything else
It was working fine in apache-airflow-providers-databricks==6.12.0, when upgraded to 7.2.1 version, started getting the error.
Issue maybe related to this MR: https://github.com/apache/airflow/pull/44960
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2738986472
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2757813077
I've recreated the exception that you're seeing, starting to implement a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2755817102
> Are there any resolutions in the short-term such as pinning an old version?
@walkeralexander, I'll take a closer look at that. But yes, I believe that could be a work around. Biggest thing would be making sure that pinning an older version doesn't create a dependency conflict elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "walkeralexander (via GitHub)" <gi...@apache.org>.
walkeralexander commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2755002952
Are there any resolutions in the short-term such as pinning an old version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "jroach-astronomer (via GitHub)" <gi...@apache.org>.
jroach-astronomer commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2754799631
I think I'd like to take a closer look at this one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2754889553
I'd love to take a look at this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "dheerajkumar-solanki (via GitHub)" <gi...@apache.org>.
dheerajkumar-solanki commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2739038747
Issue is in this function:
https://github.com/apache/airflow/blob/492ecfe5c03102bfb710108038ebd5fc50cb55b5/providers/databricks/src/airflow/providers/databricks/operators/databricks.py#L1055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "dheerajkumar-solanki (via GitHub)" <gi...@apache.org>.
dheerajkumar-solanki commented on issue #47983:
URL: https://github.com/apache/airflow/issues/47983#issuecomment-2761057793
@jroachgolf84 raised the MR for fixing the issue, issue was majorly in `_generate_databricks_task_key` function. Tried solution with dict, can you review the solution if you have not started the the development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DatabricksNotebookOperator generating invalid dependency graph [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #47983: DatabricksNotebookOperator generating invalid dependency graph
URL: https://github.com/apache/airflow/issues/47983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org