[GitHub] [airflow] allebacco commented on issue #21380: Databricks: support for triggering jobs by name

[GitBox <gi...@apache.org>]

allebacco commented on issue #21380:
URL: https://github.com/apache/airflow/issues/21380#issuecomment-1034734902


   I think that this is an interesting feature. Since there is not a direct connection between Databricks and Airflow, the id of the job could have changed due to re-creation, but the name could be more stable than the id.
   
   A possible solution could be to update the execute in this way:
   ```python
   # ... in the DatabricksRunNowOperator
   def execute(self, context):
       hook = self._get_hook()
   
       if 'job_name' in self.json:
           # If job name has been provided, find its id
           self.json['job_id'] = hook.find_job_id_by_name(self.json['job_name'])
           del self.json['job_name']
   
       self.run_id = hook.run_now(self.json)
       _handle_databricks_operator_execution(self, hook, self.log, context)
   
   # ... in the DatabricksHook
   def find_job_id_by_name(self, name) -> int:
       # List all the jobs to find the one with the specified name
       # if none or more than 1 job with the specified name, raise exception
   ```
   
   See the dbx tool (by Databricks) for an example implementation of the `find_job_id_by_name` in the https://github.com/databrickslabs/dbx/blob/main/dbx/utils/job_listing.py file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org