You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/21 09:25:06 UTC
[GitHub] [airflow] potiuk edited a comment on pull request #14915: Running tests in parallel
potiuk edited a comment on pull request #14915:
URL: https://github.com/apache/airflow/pull/14915#issuecomment-803540351
Monitoring helped a lot:
* I already decreased the use of MySQL memory from ~ 450 MB in peak to ~200MB. The default MySQL configuration we used was not optimised at all for our test usage
* However it seems that we have a serious problem in our CLI Tests that often spins out of control:
https://github.com/potiuk/airflow/runs/2158887308?check_suite_focus=true#step:6:4589
Usually our test `airflow` container max at around 400MB, but in cases where we have Exit 137 and failed tests, those spin out of control a lot a
```
Overall resource statistics
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7a4dffd2e4123a8f567704e22d13a41010c3e3f56f6759eac0c56dbb02138bea airflow-providers_airflow_run_92452ab2adba 22.43% 643MiB / 6.791GiB 9.25% 2.21MB / 2.66MB 21.7MB / 307kB 8
7605d1260aea0e05c3b5c927a43f85bfd0c89f9165e59218d9b7b5773caf84c1 airflow-providers_mysql_1 0.20% 94.01MiB / 6.791GiB 1.35% 2.66MB / 2.21MB 36.9kB / 432MB 32
88d5fd7c47ef1dc1c1269a488b2800183c262a68a3b73a6333bebff2b365e288 airflow-cli_airflow_run_17f7dab0ae27 142.86% 1.829GiB / 6.791GiB 26.93% 9.15MB / 8.32MB 15.1MB / 479kB 54
c1141763d37ed964f6d6f76e6f842c45e0bc99d5d91d323b3bd069b456c6511d airflow-cli_mysql_1 0.19% 104.8MiB / 6.791GiB 1.51% 8.32MB / 9.14MB 684kB / 467MB 38
````
See 1.829GB use of CLI airflow container
and then
```
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7a4dffd2e4123a8f567704e22d13a41010c3e3f56f6759eac0c56dbb02138bea airflow-providers_airflow_run_92452ab2adba 58.75% 633.5MiB / 6.791GiB 9.11% 2.26MB / 2.71MB 31.6MB / 307kB 8
7605d1260aea0e05c3b5c927a43f85bfd0c89f9165e59218d9b7b5773caf84c1 airflow-providers_mysql_1 4.69% 90MiB / 6.791GiB 1.29% 2.71MB / 2.26MB 1.26MB / 432MB 32
88d5fd7c47ef1dc1c1269a488b2800183c262a68a3b73a6333bebff2b365e288 airflow-cli_airflow_run_17f7dab0ae27 71.13% 3.781GiB / 6.791GiB 55.68% 9.32MB / 8.43MB 16.1MB / 483kB 42
c1141763d37ed964f6d6f76e6f842c45e0bc99d5d91d323b3bd069b456c6511d airflow-cli_mysql_1 0.05% 101.4MiB / 6.791GiB 1.46% 8.43MB / 9.32MB 684kB / 469MB 38
```
Then it gets to 3.78 GB
This is always around this state of CLI tests:
```
### The last lines for CLI process ###
tests/cli/commands/test_sync_perm_command.py . [ 67%]
tests/cli/commands/test_task_command.py ..........s...........
```
so I think it will be rather easy to find the culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org