You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hussein-awala (via GitHub)" <gi...@apache.org> on 2023/11/23 20:56:02 UTC
[PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
hussein-awala opened a new pull request, #35828:
URL: https://github.com/apache/airflow/pull/35828
Currently, the test runs with 3 parallel processes, but there are 4 tests. The last test starts when the 3 other tests finish, which leads to resources and time wasting. Running the 4 tests in parallel will help to reduce the overall time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35828:
URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824976961
Yeah... same case - a lot of "connection reset by peer" - which likely means killed by lack of memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #35828:
URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824939251
> doing it this way is good for experimenting, but not good for merging
Yes, it's just to test, then we can discuss the implementation if we find some benefits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35828:
URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824976608
> > doing it this way is good for experimenting, but not good for merging
>
> Yes, it's just to test, then we can discuss the implementation if we find some benefits.
And yeah. I would very much love to optimse these tests - they are now on critical path after all the other improvements :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35828:
URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824908372
Few comments:
Let's see. Tried that before but it was introducing a lot of random failures (likely 4 k8s simply use too much memory).
Likely we could think about switching a bit memory utilisation on our self-hosted runners (we allocate a lot of memory for tmpfs to do things more stable/faster - so whole docker installation runs using docker backed by tmpfs - not by "regular" filesystems on our self-hosted runners. Currently 85% of memory (of 64 GB) is used as tmpfs and only remaining 15% is used for regular memory of the system. Maybe some optimisations of images and size could allow us to decrease the % memory used for filesystem and then running 4 k8s will be fine ?
https://github.com/apache/airflow-ci-infra/blob/main/github-runner-ami/packer/files/mounts_setup.sh
One more comment - doing it this way is good for experimenting, but not good for merging. Hardcoding parallelism should not happen - because when it runs on self-hosted runners it's fine to run 3 but for public runners even 2 is too much - so we have special parallelism for k8s tests that has a bit weird formula to fine-tune behaviour of automated parallelism calculation
```
option_parallelism_cluster = click.option(
"--parallelism",
help="Maximum number of processes to use while running the operation in parallel for cluster operations.",
type=click.IntRange(1, max(1, (mp.cpu_count() + 1) // 3) if not generating_command_images() else 4),
default=max(1, (mp.cpu_count() + 1) // 3) if not generating_command_images() else 2,
envvar="PARALLELISM",
show_default=True,
)
```
So eventually if we want to change parallelism - we will have to change default calculation here, or have a conditional PARALLELISM setting in CI depending on type of worker (I'd very much prefer the default calculation as conditionals in GH aations CI tend to look strange and be prone to errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Increase `run-complete-tests` parallelism to 4 which is the combinations count [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed pull request #35828: Increase `run-complete-tests` parallelism to 4 which is the combinations count
URL: https://github.com/apache/airflow/pull/35828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org