[GitHub] [airflow] mharrisb1 commented on issue #24730: Google CloudRun job operator

[GitBox <gi...@apache.org>]

mharrisb1 commented on issue #24730:
URL: https://github.com/apache/airflow/issues/24730#issuecomment-1336303854

   ## Brief Thoughts/Notes
   
   I've done a little bit of work on this and here are some notes.
   
   A PR for this feature should include operators for both:
   
   - **Cloud Run services**: Used to run code that responds to web requests, or events.
   - **Cloud Run jobs**: Used to run code that performs work (a job) and quits when the work is done.
   
   [Source](https://cloud.google.com/run/docs/overview/what-is-cloud-run).
   
   It looks like https://github.com/apache/airflow/pull/27638 only include the services operator. This would be a good start but it looks like it also uses [transport](https://github.com/apache/airflow/blob/45ce866b35f824d1ee1208ce1d624203570832bc/airflow/providers/google/cloud/hooks/cloud_run.py#L23) instead of the official client. Other GCP operators (e.g. tasks and others) use the official clients so it would be best to go the same route with this one.
   
   In my mind the biggest benefit comes from the jobs operators since that would allow users who do not want to deal with/manage K8s to use Cloud Run Jobs with arbitrary containers.
   
   The Google team is great and recently released support for jobs in the official [Cloud Run Python client](https://github.com/googleapis/python-run) (see https://github.com/googleapis/python-run/pull/65) but it won't be available until [v0.5.0](https://github.com/googleapis/python-run/pull/61) with no ETA. It also currently won't build with `apache-airflow-providers-google` because of incompatible protobuf support (see https://github.com/googleapis/python-run/issues/70).
   
   I created my own plugin for this if anyone is interested in using Cloud Run Jobs (currently does not support services) before these issues are resolved and I plan to use the official client once that is ready.
   
   https://github.com/mharrisb1/airflow-google-cloud-run-plugin
   
   Please note that this plugin will only supported until this is available in Airflow.
   
   ## Requirements Proposal
   
   I think Cloud Run Services and Jobs would be great additions to GCP resources in Airflow. I think a PR to add these features should cover the following:
   
   - [ ] `CloudRunHook` - manage w/ create, read, update, delete, list resources including services, revisions, jobs, executions, and task resources (see https://github.com/googleapis/python-run/tree/main/google/cloud/run_v2/services)
   - [ ] Custom links for resources (I'm still fuzzy around what should be needed here). See other GCP resource custom link examples https://github.com/apache/airflow/tree/main/airflow/providers/google/cloud/links
   - [ ] CRUD-based operators for all resources (see task operators as an example https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/operators/tasks.py)
   - [ ] Sensors 🤷 (again, fuzzy on this one)
   
   Some additional thoughts:
   - There seems to be a pretty low quota for sequential requests so any ping mechanism should try to respect this otherwise tasks will fail often.
   - For Cloud Run jobs operators specifically, it would be nice if instead of only having CRUD-based operators, the main job run operator could also have an option to "create if not exists" and "deleted on exit" to avoid extra tasks. This is simply a personal preference (I added it to my plugin (see [example](https://github.com/mharrisb1/airflow-google-cloud-run-plugin#simple-job-lifecycle)).
   
   Would love to contribute and collaborate with anyone on this. I do think we're blocked on progress until v0.5.0 of the official client is released w/ support for compatible protofbuf lib but we can definitely go ahead and start progressing on this in preparation for those to be released in the near future (also go comment/like this issue to increase awareness to Google team https://github.com/googleapis/python-run/issues/70).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org