You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/01/23 11:16:59 UTC
[I] Use `mprocs` for `airflow standalone` when --interactive flag is passed [airflow]
potiuk opened a new issue, #60971:
URL: https://github.com/apache/airflow/issues/60971
### Description
Currently airlfow standalone multiplexes logs at the stdout, and while it is useful for non-interactive usage, when you run `airflow standalone` as an interactive user (for example when you want to do some Dag development related tasks, or even when you develop airflow itself), it's not really helpful to see those logs multiplexed.
Recently in `breeze` we switched to `mprocs` as a default terminal multiplexer and after discussion in https://lists.apache.org/thread/lj1lj2typ8mwj75pmjt3gs1o6k9p7dmf the conclusion is that `mprocs` would be a great terminal/log multiplexer for `airflow standalone --interactive` mode (new mode that we would like to add).
It has some nice featuers:
* intuitive TUI interface
* ability to control, restart and generally manage each of the sub-processes
* ability to see logs separately for each process
* intuitive keyboard and mouse navigation
* nice copy&paste support for issue reporting and diagnostics
### Use case/motivation
Interactive `airflow standalone` would be nice for anyone using `airflow standalone` to test and debug Dags.
It is also good to test and debug airflow itself - for local virtualenve development it might be a convenien,t local equivalent of `start-airflow` without the need of having docker and docker compose (with limitations of course - only working for sqlite, and only supporting local executor).
### Related issues
* https://github.com/apache/airflow/pull/58702
* https://github.com/apache/airflow/pull/58718
* https://github.com/apache/airflow/pull/60844
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use `mprocs` for `airflow standalone` when --interactive flag is passed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #60971:
URL: https://github.com/apache/airflow/issues/60971#issuecomment-3790447898
Yes. It shoud be "truly standalone".
The idea to make it totally independent from `breeze` - so that it does not require docker **at all**. Also this is both for those who want to just start airflow, try it and - for example - do some testing with their Dags, as well as for those who might want to have a nicer and quicker (but very limited) replacement for `start-airflow` - without the overhead of containers.
This means that it should not rely on mprocs being installed - it should download and install (well make it runnable internally in some cache in AIRFLOW_HOME most likely) mprocs automatically (ideally with some consensus of the user that should be asked for it unless they specify something like `--answer yes`. This is easy as mprocs publishes multi-platform binaries (it's a rust project) that are somewhat big binaries - but not **too** big and "standalone" as well - means they do not need any 3rd-party libraries. So it's just the matter of pulling and unpacking the binary that is right for the architecture we are in (on failing in case our architecture is not supported) .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use `mprocs` for `airflow standalone` when --interactive flag is passed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #60971:
URL: https://github.com/apache/airflow/issues/60971#issuecomment-3790482391
Also - Aritra volunteered on the devlist so I assigned him, But It could also bee cooperation between several people.
Also @ferruzzi mentors some people with MLH (Major League Hacking) who could do it or help with it.
It's not super complex to do, I think most of the work will be to work-out the abstraction in `airflow standalone` to be able to easily switch between "regular" and "mprocs" proces management - in case of `mprocs` - it handles a lot of things already - things like signal propagation, quitting, restarting etc.
But we already have a number of features that should be thoroughly tested and reviewed and maybe combined with the dev mode as well - to make `airflow standalone` more usable for "locally developing Airflow" case:
* hot reload - it should nicely work with `airflow standalone` in this case, but maybe we want to do some better integration (mprocs already supports server that accept commands and it could easily be used instead of built-in hot-reload - because the built-in one has some limitations (module reloading might be problematic)
* (re)building assets automatically when we start stand-alone from sources
And probably some more interesting things that can be found while implementing and playing with such `--interactive` mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use `mprocs` for `airflow standalone` when --interactive flag is passed [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #60971:
URL: https://github.com/apache/airflow/issues/60971#issuecomment-3789871618
If I understand it correctly based on the context from dev mailing list, I think it supports this out of the box, so when user `pip install apache-airflow` should be able to use `--interactive` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use `mprocs` for `airflow standalone` when --interactive flag is passed [airflow]
Posted by "Prab-27 (via GitHub)" <gi...@apache.org>.
Prab-27 commented on issue #60971:
URL: https://github.com/apache/airflow/issues/60971#issuecomment-3789814437
@potiuk Is it fine to run the tests in the Docker environment (Breeze), or is it necessary to run them in a virtual environment (venv)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org