[D] Support for SkyPilot provider in Airflow [airflow]

["kevinmingtarja (via GitHub)" <gi...@apache.org>]

GitHub user kevinmingtarja created a discussion: Support for SkyPilot provider in Airflow

### Description

Hi! We are the team behind [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running AI and batch workloads across any infra. We have an existing guide on how to orchestrate SkyPilot tasks using Airflow: https://docs.skypilot.co/en/latest/examples/orchestrators/airflow.html

But it involves the user writing a lot of boilerplate code just to run a simple SkyPilot task. Recently, we've been getting some requests to simplify this process, so we thought it would be beneficial to package the logic there as an Airflow operator.

### Use case/motivation

Some of our users use Airflow for orchestrating SkyPilot tasks. So they would define their jobs with SkyPilot (i.e. running a python script for data preprocessing) and SkyPilot would handle the provisioning of nodes, environment setup, and the running of the jobs. And airflow would handle the scheduling (i.e. make it recurring every 12h), data flow between different tasks, logging, monitoring, etc.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/53925

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Support for SkyPilot provider in Airflow [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Support for SkyPilot provider in Airflow

The current process for contributing the provider to be "community provider" is described here https://github.com/apache/airflow/blob/main/PROVIDERS.rst#community-managed-providers - please read the documentation there including some past links to discussion and when we decided to accept or not a 3rd-party contributed provider. 

There are basically two ways you can follow:

1) You can release provider on your own - independently of the community
2) or - if you convince the community that this is fine for the community to accept the provider and maintenance of it (following the criteria describe in the document I linked) - it could become managed in the Airflow monorepo.


Generally speaking - any new provider must be discussed on the devlist and there should be either LAZY CONSENSUS reached or vote passed to accept it.

I am converting it to a discussion if needed, but you should generally follow the docs and past discussions on it.



GitHub link: https://github.com/apache/airflow/discussions/53925#discussioncomment-13933977

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Support for SkyPilot provider in Airflow [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

GitHub user boring-cyborg[bot] added a comment to the discussion: Support for SkyPilot provider in Airflow

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


GitHub link: https://github.com/apache/airflow/discussions/53925#discussioncomment-13933976

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org