[GitHub] [airflow] jaketf opened a new issue, #23337: Suspend Airflow API endpoint

[GitBox <gi...@apache.org>]

jaketf opened a new issue, #23337:
URL: https://github.com/apache/airflow/issues/23337

   ### Description
   
   add a global `/suspend` and `/resume` endpoint to the REST API (and corresponding CLI command) that would do the following for the entire airflow deployment:
   
   `/suspend`
   - pause all currently unpaused DAGs (keeping track of what was previously unpaused in the metadata db)
   - Set `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE=False` (if not already false) to prevent scheduler from scheduling more tasks
   - (should handle duplicate suspend calls by `409: Airflow is already suspended`)
   
   `/resume`
   - unpause all DAGs from a previous `/suspend` call
   - set `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE` to whatever it was _before_ the last `/suspend` call
   - (should handle resuming an airflow that is not in the suspended state with `409: Cannot resume an airflow that is not in suspended state`)
   
   ### Use case/motivation
   
   This could be really helpful for people wanting to pause everything:
   - before running and upgrade to a newer airflow version
   - because this is a dev enviornment and we don't need anything to run outside of working hours to reduce resource consumption
   - "archiving an airflow" call suspend then tear down all the airflow components, could be restored by bringing up the components and then calling resume.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23337: Suspend Airflow API endpoint

[GitBox <gi...@apache.org>]

potiuk commented on issue #23337:
URL: https://github.com/apache/airflow/issues/23337#issuecomment-1120448250

   literally setting `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE` via API is not possible, because we would have to a) change it in env variables for running process (because it is impossible) b) do it in scheduler - and the API/webserver does not have direct way to communicate with scheduler, it would have to be done through database rather than through env variables. 
   
   I am not sure if this is a good idea to influence scheduler's behaviour via API. I'd say any cahnge of scheduler behaviour should be done via  deployment. What we COULD do instead we could have Airflow scheduler to respond to (for example) USR1 signal and have a way to pass it via the deployment).
   
   Converting it into discussion - maybe someone will have other ideas. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org