You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/24 00:32:16 UTC
[GitHub] [airflow] kaxil opened a new pull request #16623: Set Process title for Worker when using ``LocalExecutor``
kaxil opened a new pull request #16623:
URL: https://github.com/apache/airflow/pull/16623
This has annoyed me for a long time. When using ``LocalExecutor``, it was difficult to see which process is a worker as it just showed up as below -- which had same title as parent scheduler process. This PR/commit adds a title for idle workers and when a task is running it has the "command" that is running in the title, similar to our supervising process
Before:
```
root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash
root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver
root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver]
root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver]
root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash
root 13933 31.0 0.9 466596 117656 pts/2 S+ 00:22 0:01 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13941 0.0 0.7 466340 97988 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13942 3.2 0.8 1392072 100136 pts/2 Sl+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13950 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13952 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13955 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13958 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13962 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13966 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13969 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13975 0.0 0.8 466340 98404 pts/2 S+ 00:22 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13979 6.5 0.8 466596 99956 pts/2 S 00:22 0:00 \_ airflow scheduler -- DagFileProcessorManager
```
After (with no running tasks - idle workers):
```
root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash
root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver
root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver]
root 1587 0.6 2.5 985752 318184 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver]
root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash
root 13237 25.7 0.9 466596 117692 pts/2 S+ 00:20 0:02 \_ airflow worker -- LocalExecutor
root 13245 0.1 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13246 2.1 0.8 1318340 100104 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13254 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13256 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13259 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13263 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13267 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13271 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13282 4.1 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager
```
After (with running tasks):
```
root@a7c8aa590704:/opt/airflow# ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6434 0.0 0.0 6652 4584 pts/3 Ss 00:01 0:00 /bin/bash
root 13622 0.0 0.0 9396 3032 pts/3 R+ 00:21 0:00 \_ ps auxf
root 1 0.0 0.0 2148 720 ? Ss Jun23 0:00 /usr/bin/dumb-init -- /entrypoint
root 7 0.0 0.0 6656 4400 pts/0 Ss Jun23 0:00 /bin/bash
root 121 0.0 0.0 8220 3228 pts/0 S+ Jun23 0:00 \_ tmux
root 101 0.0 0.0 15856 4272 ? Ss Jun23 0:00 /usr/sbin/sshd
root 123 0.0 0.0 10184 5012 ? Ss Jun23 0:00 tmux
root 124 0.0 0.0 6676 4636 pts/1 Ss Jun23 0:00 \_ -bash
root 1449 0.8 2.6 988356 326312 pts/1 Sl+ Jun23 0:16 | \_ /usr/local/bin/python /usr/local/bin/airflow webserver
root 1584 0.0 0.4 121068 56864 pts/1 S+ Jun23 0:01 | \_ gunicorn: master [airflow-webserver]
root 1587 0.6 2.5 986144 318712 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1588 0.6 2.5 984776 317672 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1589 0.6 2.5 985688 318148 pts/1 Sl+ Jun23 0:12 | \_ [ready] gunicorn: worker [airflow-webserver]
root 1590 0.6 2.5 985200 317776 pts/1 Sl+ Jun23 0:11 | \_ [ready] gunicorn: worker [airflow-webserver]
root 128 0.0 0.0 6676 4552 pts/2 Ss Jun23 0:00 \_ -bash
root 13237 5.8 0.9 467512 118572 pts/2 S+ 00:20 0:03 \_ airflow worker -- LocalExecutor
root 13245 0.0 0.7 466340 97804 pts/2 S+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13246 0.5 0.8 1318340 100108 pts/2 Sl+ 00:20 0:00 \_ /usr/local/bin/python /usr/local/bin/airflow scheduler
root 13254 0.0 0.8 466340 98976 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_0', '2021-06-24T00:21:31.387956+00:00', '--local', '
root 13256 0.0 0.8 466340 98980 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_1', '2021-06-24T00:21:31.387956+00:00', '--local', '
root 13259 0.0 0.8 466340 98980 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'runme_2', '2021-06-24T00:21:31.387956+00:00', '--local', '
root 13263 0.0 0.8 466340 98980 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'also_run_this', '2021-06-24T00:21:31.387956+00:00', '--loc
root 13267 0.0 0.8 466340 98980 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'this_will_skip', '2021-06-24T00:21:31.387956+00:00', '--lo
root 13271 0.0 0.8 466340 98980 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor: ['airflow', 'tasks', 'run', 'example_bash_operator', 'run_after_loop', '2021-06-24T00:21:31.387956+00:00', '--lo
root 13274 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13276 0.0 0.8 466340 98396 pts/2 S+ 00:20 0:00 \_ airflow worker -- LocalExecutor
root 13282 1.2 0.8 466596 99952 pts/2 S 00:20 0:00 \_ airflow scheduler -- DagFileProcessorManager
root@a7c8aa590704:/opt/airflow#
```
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #16623: Set Process title for Worker when using ``LocalExecutor``
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #16623:
URL: https://github.com/apache/airflow/pull/16623#issuecomment-867476504
@kaxil I've set this to 2.1.2 -- 2.1.1. is in feature freeze now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #16623: Set Process title for Worker when using ``LocalExecutor``
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #16623:
URL: https://github.com/apache/airflow/pull/16623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16623: Set Process title for Worker when using ``LocalExecutor``
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16623:
URL: https://github.com/apache/airflow/pull/16623#issuecomment-867439715
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org