You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/30 11:44:10 UTC
[GitHub] [airflow] potiuk opened a new pull request #16169: Mark `test_send_tas_to_celery_hang` as quarantined
potiuk opened a new pull request #16169:
URL: https://github.com/apache/airflow/pull/16169
The test_send_tasks_to_celery_hang hangs on self-hosted runners more
often than not.
It's been introduced in #15989 and while the test does not usually hang
on regular GitHub runners, or in case of running it locally (I could not
make it fail), it does hang almost always when run on self-hosted
runners.
Marking it as quarantined for now.
Issue #16168 created to keep track of it.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yuqian90 commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851849148
> I added the link and copied the stack-trace: @yuqian90 in #16168. We can observe if it is happening in master builds (they always run on self-hosted infra) in the Quarantined job.
Hi, @potiuk thanks for the info. Unfortunately I don't have a convenient way to reproduce the environment on my end. I tried to run `test_send_tas_to_celery_hang` thousands of times on the debian, ubuntu and osx machines I have access to, it never hanged. The test is passing through the same code path that celery_executor would be using if it were to run in production. I'm wondering if celery_executor hangs in this environment too with or without the fix in #15989 .
Worst case, is there a way to make `test_send_tas_to_celery_hang` only run when it's not in the self-hosted runner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851014021
> Thanks @potiuk. Do you have any details for these "self-hosted runners"? E.g what OS are they running on and using what version of python?
Yeah. Sorry I have not added the links. I update the description in a moment. It seems for all versions of python. The system is based on ubuntu - here are all the `packer` scripts that prepare the runner instance: https://github.com/apache/airflow-ci-infra/tree/main/github-runner-ami/packer
What is different for those instances is that they are much bigger than runner (64GB RAM, 8VCPUS vs 6GB/2VCPUs) and what we implemented there is that most of the flesystem used is tmpfs-backed:
```
sudo mount -t tmpfs -o size=10% tmpfs /tmp
sudo mount -t tmpfs -o size=66% tmpfs /var/lib/docker
sudo mount -t tmpfs -o tmpfs /home/runner/actions-runner/_work
```
That makes everything much faster and has much less chance of being synchronized on whatever file acccess.. I presume that might be the biggest reason for observed differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851015296
I added the link and copied the stack-trace: @yuqian90 in #16168. We can observe if it is happening in master builds (they always run on self-hosted infra) in the Quarantined job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851853976
I think we need to stabilize the CI self-hosted runners a bit - we need to (very soon) switch to GCP-hosted ones and there are some instabilities currently due to the way it is run for auto-scaling which we will fix. Not sure if this is related but I think I only saw it in few cases. The quarantined tests will be running with every master merge so we can take a look if the situation re-occurs (and we can move it out of quarantine then).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851854371
I assigned both of us to that for now. I will take a look and will keep you posted on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #16169:
URL: https://github.com/apache/airflow/pull/16169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yuqian90 commented on pull request #16169: Mark `test_send_tasks_to_celery_hang` as quarantined
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on pull request #16169:
URL: https://github.com/apache/airflow/pull/16169#issuecomment-851002248
Thanks @potiuk. Do you have any details for these "self-hosted runners"? E.g what OS are they running on and using what version of python?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org