You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/13 19:23:42 UTC
[GitHub] [airflow] denimalpaca opened a new issue, #24422: Add SQL Provider
denimalpaca opened a new issue, #24422:
URL: https://github.com/apache/airflow/issues/24422
### Description
Adds a new SQL Provider for new SQL Check Operators.
### Use case/motivation
This will allow for faster development and maintenance of non-core SQL operators.
### Related issues
#23874 Add Templated SQL Check Operators for Columns and Tables
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1156325336
If so it may speed up https://github.com/apache/airflow/pull/23971 as it will not be depended on Airflow core version but on a provider version.
As for the provider name i just wondered if it should be provider sql or provider core with sql as extra. Just for the case where we might want to extract other core operators/hooks/sensors into provider(?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1156330932
> As for the provider name i just wondered if it should be provider sql or provider core with sql as extra. Just for the case where we might want to extract other core operators/hooks/sensors into provider(?)
Interesting idea, however extras are "evil". I'd say `apache-airflow-providers-core-sql` is better approach.
This will allow independent release of similar features - we avoid the fate of "google" provider this way, where things that are "loosely" related (i.e. by google) will be actually coupled when it comes to releasing them. I can easily imagine if we have (for example) `core-storage' and `core-sql` that we would like to release only "core-sql` with a new feature, without touching 'core-storage`..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1154884356
I think in the future this might become an interesting pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1154897948
Do you plan to move `DbApiHook` there?
Do you see it as sql provider or core provider with sql as standalone option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] denimalpaca commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
denimalpaca commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1155207481
I can't really speak to `DbApiHook`, as I'm not sure what all the consequences may be. Not sure what a core provider is relative to other providers, but I see this as a SQL provider that would be installed with Airflow (is that what you mean by core provider?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1154845477
Why do we need provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1154883605
This is the proposal I mentioned in #23874. Apparently the Data Lineage effort will require more "SQL-bound" common functionality. Adding this "common functionality" in Airflow, is not very good, because it will only be released in 2.4 earliest, and it means that providers will be able to depend on those 12 months after.
Having a common "SQL" provider which keeps such common code that other providers might depend on, is the way how to make such "common sql" functionality available for multiple providers instantly. We should just make dependency from (say) Presto, Postgres, MySQL , Trino, .... providers to the "sql" provider and we will be able to install and use those providers and the common function on Airflow 2.2 immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1168462283
Merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24422: Add SQL Provider
URL: https://github.com/apache/airflow/issues/24422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1156241866
I think it would make sense to move DBAPI there too and all related functions. There is no particular reasons why DBApi should be part of the "core". We could deprecate it and move it all to provider, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24422: Add SQL Provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24422:
URL: https://github.com/apache/airflow/issues/24422#issuecomment-1156356871
yes exactly like we have now for example with Microsoft.
The provider is Microsoft but azure and mssql are totally separated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org