You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dimonchik-suvorov (via GitHub)" <gi...@apache.org> on 2023/06/14 13:01:25 UTC
[GitHub] [airflow] dimonchik-suvorov opened a new issue, #31898: DatabricksSqlHook fails to run related queries
dimonchik-suvorov opened a new issue, #31898:
URL: https://github.com/apache/airflow/issues/31898
### Apache Airflow version
main (development)
### What happened
After refactoring `DatabricksSqlHook` lost it's ability to run multiple queries in one Airflow connection (basically same Spark Session). Now it doesn't care what the smart [get_conn](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks_sql.py#L96) returns and simply creates new connection for each query in the queries list. It is wrong and breaks logic like:
```
USE your_database;
select from table_in_your_database;
```
In this case it will create first connection and as consequence Databricks session, runs `USE your_database` statement, then recreate connection/session which doesn't know anything about first query and runs `select from table_in_your_database` statement and obviously it fails because second session doesn't know anything about first query...
<img width="1543" alt="image" src="https://github.com/apache/airflow/assets/23456894/284b87a7-a183-4e99-b8f5-b7afe1a10036">
This is a breaking change and cost us some time to understand what went wrong after our upgrade from `2.2.0` to `2.5.3`
### What you think should happen instead
smart [get_conn](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks_sql.py#L96) functions should decide when to create new connection and when not to. Otherwise why do we need so complex logic there?
### How to reproduce
Try to send (using `DatabricksSqlHook`) to the Databricks Warehouse a string with two queries:
```
USE your_database;
select from table_in_your_database;
```
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
`databricks 3.4.0`
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #31898: DatabricksSqlHook fails to run related queries
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #31898: DatabricksSqlHook fails to run related queries
URL: https://github.com/apache/airflow/issues/31898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #31898: DatabricksSqlHook fails to run related queries
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31898:
URL: https://github.com/apache/airflow/issues/31898#issuecomment-1591969067
fixed in https://github.com/apache/airflow/issues/31898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org