You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/07 10:23:25 UTC
[GitHub] [airflow] dinowernli opened a new issue #21380: Databricks: support for triggering jobs by name
dinowernli opened a new issue #21380:
URL: https://github.com/apache/airflow/issues/21380
### Description
The DatabricksRunNowOperator supports triggering job runs by job ID. We would like to extend the operator to also support triggering jobs by name. This will likely require first making an API call to list jobs in order to find the appropriate job id.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21380: Databricks: support for triggering jobs by name
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21380:
URL: https://github.com/apache/airflow/issues/21380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eskarimov commented on issue #21380: Databricks: support for triggering jobs by name
Posted by GitBox <gi...@apache.org>.
eskarimov commented on issue #21380:
URL: https://github.com/apache/airflow/issues/21380#issuecomment-1035979552
Indeed, it'd be a very useful feature. May I work on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] allebacco commented on issue #21380: Databricks: support for triggering jobs by name
Posted by GitBox <gi...@apache.org>.
allebacco commented on issue #21380:
URL: https://github.com/apache/airflow/issues/21380#issuecomment-1034734902
I think that this is an interesting feature. Since there is not a direct connection between Databricks and Airflow, the id of the job could have changed due to re-creation, but the name could be more stable than the id.
A possible solution could be to update the execute in this way:
```python
# ... in the DatabricksRunNowOperator
def execute(self, context):
hook = self._get_hook()
if 'job_name' in self.json:
# If job name has been provided, find its id
self.json['job_id'] = hook.find_job_id_by_name(self.json['job_name'])
del self.json['job_name']
self.run_id = hook.run_now(self.json)
_handle_databricks_operator_execution(self, hook, self.log, context)
# ... in the DatabricksHook
def find_job_id_by_name(self, name) -> int:
# List all the jobs to find the one with the specified name
# if none or more than 1 job with the specified name, raise exception
```
See the dbx tool (by Databricks) for an example implementation of the `find_job_id_by_name` in the https://github.com/databrickslabs/dbx/blob/main/dbx/utils/job_listing.py file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21380: Databricks: support for triggering jobs by name
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21380:
URL: https://github.com/apache/airflow/issues/21380#issuecomment-1036253999
Feel free!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org