You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/11 23:34:56 UTC

[GitHub] [airflow] Gollum999 opened a new issue #21531: --log-file/--stdout/--stderr options ignored

Gollum999 opened a new issue #21531:
URL: https://github.com/apache/airflow/issues/21531


   ### Apache Airflow version
   
   2.2.3 (latest released)
   
   ### What happened
   
   Some airflow processes (I've tested `webserver`, `scheduler`, `triggerer`, `celery worker`, and `celery flower`) ignore the `--log-file`, `--stdout`, and `--stderr` options when running without `--daemon`.  No log files are created, and logs continue being printed to stdout/stderr.  This can be a bit confusing in environments where daemonization is handled externally, e.g. when running through [systemd](https://airflow.apache.org/docs/apache-airflow/stable/howto/run-with-systemd.html).
   
   When `--daemon` is added, most process types will suppress stdout/stderr and write to the specified log files as expected.  **However, `webserver` does not write any logs except for a few lines on shutdown, and since stdout/stderr are suppressed, all useful logs are lost.**  (Though worth noting that `--access-logfile` and `--error-logfile` work as expected, with or without `--daemon`.)
   
   ### What you expected to happen
   
   Command line args should be respected whenever possible.  If there are certain combinations of args that are incompatible, they should be noted as such in the `--help` and/or emit a warning at runtime.
   
   With `airflow webserver` in particular, these arguments seem to be mostly useless since no (useful) logs are written to a file, regardless of whether or not `--daemon` is specified.  If the webserver behavior cannot be changed, maybe these options should just be removed.
   
   ### How to reproduce
   
   With a fresh install (no changes to `airflow.cfg`, etc):
   
   ```
   airflow webserver --log-file=$AIRFLOW_HOME/webserver.log --stdout=$AIRFLOW_HOME/webserver.out --stderr=$AIRFLOW_HOME/webserver.err  # no logs created, stdout/stderr not redirected
   airflow webserver --log-file=$AIRFLOW_HOME/webserver.log --stdout=$AIRFLOW_HOME/webserver.out --stderr=$AIRFLOW_HOME/webserver.err --daemon  # logs created but remain empty until webserver is killed, stdout/stderr suppressed after startup
   kill $(cat $AIRFLOW_HOME/airflow-webserver-monitor.pid)
   
   airflow scheduler --log-file=$AIRFLOW_HOME/scheduler.log --stdout=$AIRFLOW_HOME/scheduler.out --stderr=$AIRFLOW_HOME/scheduler.err  # no logs created, stdout/stderr not redirected
   airflow scheduler --log-file=$AIRFLOW_HOME/scheduler.log --stdout=$AIRFLOW_HOME/scheduler.out --stderr=$AIRFLOW_HOME/scheduler.err --daemon  # seems to work as expected
   kill $(cat $AIRFLOW_HOME/airflow-scheduler.pid)
   
   # Repeat as necessary for other process types
   ```
   
   ### Operating System
   
   CentOS 7.4
   
   ### Versions of Apache Airflow Providers
   
   N/A
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Just running locally, no containers.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kadai0308 edited a comment on issue #21531: --log-file/--stdout/--stderr options ignored

Posted by GitBox <gi...@apache.org>.
kadai0308 edited a comment on issue #21531:
URL: https://github.com/apache/airflow/issues/21531#issuecomment-1046451226


   Hi @potiuk, nice to meet you. I am Kadai, new here.
   
   For the webserver part, after go through the code, for writing the log when daemon is off,  is it just write the output to a file at [here](https://github.com/apache/airflow/blob/main/airflow/cli/commands/webserver_command.py#L475) ?
   
   something like:
   ```
   with subprocess.Popen(run_args, close_fds=True, stdout=out_logfile, stderr=error_logfile) as gunicorn_master_proc:
   ```
   Not sure am i missing anything else need to modify, if not, then i think i can help on this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #21531: --log-file/--stdout/--stderr options ignored

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21531:
URL: https://github.com/apache/airflow/issues/21531#issuecomment-1052681719


   I am not sure if understand. I will convert it into discussion and maybe the @Gollum999 and @kadai0308 you can can make a proposal what to do with it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #21531: --log-file/--stdout/--stderr options ignored

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21531:
URL: https://github.com/apache/airflow/issues/21531#issuecomment-1036795271


   I think it would be great to update the help on that. PR to clarify the behaviours are most welcome. Would you like to update the help as appropriate? You can easily become one of the almost 2000 contributors to airflow ? This seems like a cool contribution to review and update the messages related to those flags. I think, since you just experienced it, you are one of the best people to do it @Gollum999. The PRs for that should be rather easy.
   
   The right descriptions/help messages can be found here https://github.com/apache/airflow/tree/main/airflow/cli/commands 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kadai0308 edited a comment on issue #21531: --log-file/--stdout/--stderr options ignored

Posted by GitBox <gi...@apache.org>.
kadai0308 edited a comment on issue #21531:
URL: https://github.com/apache/airflow/issues/21531#issuecomment-1046451226


   Hi @potiuk, nice to meet you. I am Kadai, new here.
   
   For the webserver part, after go through the code, for writing the log when daemon is off,  is it just write the output to a file at [here](https://github.com/apache/airflow/blob/main/airflow/cli/commands/webserver_command.py#L475) ?
   
   something like:
   ```
   with subprocess.Popen(run_args, close_fds=True, stdout=out_logfile, stderr=error_log)file) as gunicorn_master_proc:
   ```
   Not sure am i missing anything else need to modify, if not, then i think i can help on this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kadai0308 commented on issue #21531: --log-file/--stdout/--stderr options ignored

Posted by GitBox <gi...@apache.org>.
kadai0308 commented on issue #21531:
URL: https://github.com/apache/airflow/issues/21531#issuecomment-1046451226


   Hi @potiuk, nice to meet you. I am Kadai, new here.
   
   For the webserver part, after go through the code, for writing the log when daemon is off,  is it just write the output to a file at [here] ?(https://github.com/apache/airflow/blob/main/airflow/cli/commands/webserver_command.py#L475)
   
   something like:
   ```
   with subprocess.Popen(run_args, close_fds=True, stdout=out_logfile, stderr=error_log)file) as gunicorn_master_proc:
   ```
   Not sure am i missing anything else need to modify, if not, then i think i can help on this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org