You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/04 19:52:21 UTC
[GitHub] [airflow] Mokubyow opened a new pull request #19412: Add databricks_api_version hook and operators
Mokubyow opened a new pull request #19412:
URL: https://github.com/apache/airflow/pull/19412
This PR updates the Databricks API version from 2.0 to 2.1. Of the nine available endpoints in the DatabricksHook, these seven are not in use and do not map to any operators.
```
[NOT IN USE] api/2.1/clusters/restart => None
[NOT IN USE] api/2.1/clusters/start => None
[NOT IN USE] api/2.1/clusters/delete => None
[NOT IN USE] api/2.1/jobs/runs/get => None
[NOT IN USE] api/2.1/jobs/runs/cancel => None
[NOT IN USE] api/2.1/libraries/install => None
[NOT IN USE] api/2.1/libraries/uninstall => None
```
The remaining two endpoints map to a single operator each and the request structure is either unchanged or compatible when moving from version 2.0 to 2.1 according to this doc https://docs.databricks.com/data-engineering/jobs/jobs-api-updates.html#api-client-guide
```
[COMPATIABLE] api/2.1/jobs/runs/submit => DatabricksSubmitRunOperator
[UNCHANGED] api/2.1/jobs/run-now => DatabricksRunNowOperator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-963670237
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] christophergrant edited a comment on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
christophergrant edited a comment on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-966533359
Any APIs that are not under the `api/2.x/jobs` prefix do not have version 2.1 at this time i.e the clusters APIs and libraries APIs will need to be reverted to 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mokubyow commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
Mokubyow commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-963264031
@potiuk Yes, the two endpoints we use are fully backward compatible/unchanged as noted in the documentation I linked.
@alexott Happy to help with adding that support on another PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#discussion_r744234436
##########
File path: airflow/providers/databricks/CHANGELOG.rst
##########
@@ -19,6 +19,13 @@
Changelog
---------
+2.0.3
+.....
+
+Misc
+~~~~
+ * ``Update API version from 2.0 to 2.1``
+
Review comment:
```suggestion
```
Change log is automatically created by release manager.
No need to modify this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#discussion_r744234436
##########
File path: airflow/providers/databricks/CHANGELOG.rst
##########
@@ -19,6 +19,13 @@
Changelog
---------
+2.0.3
+.....
+
+Misc
+~~~~
+ * ``Update API version from 2.0 to 2.1``
+
Review comment:
```suggestion
```
Change log is automatically created by release manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-961379485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-962670471
I agree. And it will makes sense to wait a bit - there will be improvements in the near future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] christophergrant commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
christophergrant commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-966533359
Nit: any APIs that are not under the `api/2.x/jobs` prefix do not have version 2.1 at this time i.e the clusters APIs and libraries APIs will need to be reverted to 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-962591250
Yeah @Mokubyow - can you please remove the changelog entry ?
Also - question (just to confirm) - the 2.1 API is fully backwards compatible with 2.0 in the way we use it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-963369070
Some statuc checks failing @Mokubyow - I recommend using pre-commit to fix them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19412:
URL: https://github.com/apache/airflow/pull/19412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-961379485
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mokubyow commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
Mokubyow commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-963626497
@potiuk Looks like the pre-commit worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
alexott commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-962659502
It would be nice to add support for multi-task jobs which is primary reason for 2.1 API... Although maybe it makes sense to wait until for a new functionality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19412: Update Databricks API from 2.0 to 2.1
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19412:
URL: https://github.com/apache/airflow/pull/19412#issuecomment-962669521
> It would be nice to add support for multi-task jobs which is primary reason for 2.1 API... Although maybe it makes sense to wait until for a new functionality
Yeah. Let's split it to separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org