You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/08 11:00:52 UTC
[GitHub] [airflow] dzamo opened a new pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
dzamo opened a new pull request #16884:
URL: https://github.com/apache/airflow/pull/16884
This PR adds a new provider to Airflow for executing SQL queries against an Apache Drill instance. This is useful because, while its primary use case is interactive analysis that often avoids the need for ETL, Drill's CTAS statements and support for querying a large variety of data sources also make it a very capable ETL tool. What is lacking to use Drill this way is worklow management, and that gap is one that Airflow fills perfectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-876342968
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzamo commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
dzamo commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-876357625
I see I need to get the v1.1.0 sqlalchemy-drill release pushed to PyPI to proceed. I'll do that and trigger another build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-877670132
Take a look now at the build docs errors /static checks errors, I think they tell exactly what to do :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#discussion_r667500879
##########
File path: airflow/providers/apache/drill/hooks/drill.py
##########
@@ -0,0 +1,89 @@
+#
Review comment:
Can you create `__init__.py` file in hooks directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #16884:
URL: https://github.com/apache/airflow/pull/16884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-877625884
Some static checks + docs failing. For static checks I recommend installing `pre-commit` - it should fix the problems automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-876377146
> I see I need to get the v1.1.0 sqlalchemy-drill release pushed to PyPI to proceed. I'll do that and trigger another build.
Yep :). Glad to see another apache-family provider added :). I will take a look shortly. It looks cool and might be just-in-time for the next wave of providers (next week likely)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-877823206
The two errors now are not obvious indeed:
```
------------------------------ Error 1 --------------------
/opt/airflow/docs/apache-airflow-providers-apache-drill/_api/drill/index.rst: WARNING: document isn't included in any toctree
```
Here, i think (@mik-laj ?) "drill" has to be added to this list:
https://github.com/apache/airflow/blob/d3f300fba8c252cac79a1654fddb91532f44c656/docs/conf.py#L537
```
------------------------------ Error 2 --------------------
/opt/airflow/docs/apache-airflow-providers-apache-drill/commits.rst: WARNING: document isn't included in any toctree
```
Not obvioius, I know, but copy this below the "AIUTOGENERATED" comment in index.rst (see https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow-providers-airbyte/index.rst).
I will improve it in the future when someone adds a new provider so that this is not misleading:
```
.. toctree::
:maxdepth: 1
:caption: Commits
Detailed list of commits <commits>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-877823206
The two errors now are not obvious indeed:
```
------------------------------ Error 1 --------------------
/opt/airflow/docs/apache-airflow-providers-apache-drill/_api/drill/index.rst: WARNING: document isn't included in any toctree
```
Here, i think (@mik-laj ?) "drill" has to be added to this list:
https://github.com/apache/airflow/blob/d3f300fba8c252cac79a1654fddb91532f44c656/docs/conf.py#L537
```
------------------------------ Error 2 --------------------
/opt/airflow/docs/apache-airflow-providers-apache-drill/commits.rst: WARNING: document isn't included in any toctree
```
Not obvioius, I know, but copy this below the "AIUTOGENERATED" comment. I will improve it in the future when someone adds a new provider so that this is not misleading:
```
.. toctree::
:maxdepth: 1
:caption: Commits
Detailed list of commits <commits>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-878479165
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-878478693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzamo commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
dzamo commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-877649744
Thanks for the help @potiuk. I've applied pre-commit and added a bit to the docs while I was investigating the error there. I fear I haven't yet actually resolved the docs problem, "doc not present in any toctree", but I'm still a little puzzled about what more I might need to add, and also by what gets generated and what is written by hand. So I've pushed the new commits to see what the error messages look like now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzamo commented on pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
dzamo commented on pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#issuecomment-878208133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzamo commented on a change in pull request #16884: AIRFLOW-5529 Add Apache Drill provider.
Posted by GitBox <gi...@apache.org>.
dzamo commented on a change in pull request #16884:
URL: https://github.com/apache/airflow/pull/16884#discussion_r667693586
##########
File path: airflow/providers/apache/drill/hooks/drill.py
##########
@@ -0,0 +1,89 @@
+#
Review comment:
Ah, how did I miss that? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org