You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/16 23:39:19 UTC
[GitHub] [airflow] khalidmammadov opened a new pull request #19633: Fix failing CI phase with unhealthy container issue
khalidmammadov opened a new pull request #19633:
URL: https://github.com/apache/airflow/pull/19633
This is to fix failing Integration tests with below error that can be seen in CI builds for most PRs.
` ERROR: for airflow Container "f149d1742c3f" is unhealthy.`
It happens due to **openldap** integration. When container is started it expects some empty directories for ldap database and finds some data and fails the container entering **unhealthy** state. ([similar issue](https://github.com/osixia/docker-openldap/issues/326))
This directories are mapped from docker-compose and provided externally using volumes.
PR adds removal of integration related volumes as well prior executing of main docker-compose build.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971467056
Ahh. I see the problem. Great finding! Kudos! Just wondering why it started to happen only recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#discussion_r752399137
##########
File path: scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh
##########
@@ -123,8 +124,10 @@ function run_airflow_testing_in_docker() {
"${DOCKER_COMPOSE_LOCAL[@]}" \
--project-name "airflow-${TEST_TYPE}-${BACKEND}" \
run airflow "${@}"
+ docker ps
Review comment:
Oh yeah. That was a sneaky attempt to make our tests green again :scream: . Yeah. My bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19633:
URL: https://github.com/apache/airflow/pull/19633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971544048
Strange, but it looks like it works now :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971593257
Yep. That was it - pinning pinot helped. I also fixed in #19645 and #19646 some fixes to CI scripts to help with such investigations in the future (we uploaded container logs on failure in the past, but apparently introducing more care about memory used cause the containers removal before the logs were dumped :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on a change in pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on a change in pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#discussion_r752259727
##########
File path: scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh
##########
@@ -123,8 +124,10 @@ function run_airflow_testing_in_docker() {
"${DOCKER_COMPOSE_LOCAL[@]}" \
--project-name "airflow-${TEST_TYPE}-${BACKEND}" \
run airflow "${@}"
+ docker ps
Review comment:
I will file a fix (remove that) if you are working on it yet @ashb
@potiuk are you ok with this removal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971468495
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on a change in pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on a change in pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#discussion_r752259727
##########
File path: scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh
##########
@@ -123,8 +124,10 @@ function run_airflow_testing_in_docker() {
"${DOCKER_COMPOSE_LOCAL[@]}" \
--project-name "airflow-${TEST_TYPE}-${BACKEND}" \
run airflow "${@}"
+ docker ps
Review comment:
I will file a fix (remove that) if you are not working on it yet @ashb
@potiuk are you ok with this removal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971596321
> Thanks for the investigation @khalidmammadov !
no problems. Glad it's fixed finally!
PS. I think we forgot to remove `docker ps` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971467056
Ahh. I see the problem. Great finding! Kudos! Just wondering why it start to happen only recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971546834
Not really. The problem is that DEBUG apparently causes exit code = 0 even if it failed:
Looks like `pinot` is the guilty one (and that could explain why now - as we are using "latest" image for it.
```
ERROR: for airflow Container "86de6cfc0749" is unhealthy.
Encountered errors while bringing up the project.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
26d4322f3d1d rabbitmq:3.7 "docker-entrypoint.s…" 20 seconds ago Up 14 seconds (health: starting) 4369/tcp, 5671-5672/tcp, 25672/tcp airflow-integration-postgres_rabbitmq_1
f960bc850b74 ghcr.io/apache/airflow-trino:359-2021.07.04 "/entrypoint.sh /usr…" 20 seconds ago Up 14 seconds (health: starting) 0.0.0.0:37778->7778/tcp, :::37778->7778/tcp, 0.0.0.0:38080->8080/tcp, :::38080->8080/tcp airflow-integration-postgres_trino_1
44c71fc6ce77 apache/airflow:airflow-statsd-exporter-2020.09.05-v0.17.0 "/bin/statsd_exporter" 20 seconds ago Up 15 seconds (health: starting) 0.0.0.0:9125->9125/tcp, 0.0.0.0:9125->9125/udp, :::9125->9125/tcp, :::9125->9125/udp, 0.0.0.0:29102->9102/tcp, :::29102->9102/tcp airflow-integration-postgres_statsd-exporter_1
639198fca04c grafana/grafana "/run.sh" 20 seconds ago Up 15 seconds 0.0.0.0:23000->3000/tcp, :::23000->3000/tcp airflow-integration-postgres_grafana_1
58566c9414dc postgres:9.6 "docker-entrypoint.s…" 20 seconds ago Up 15 seconds (healthy) 5432/tcp airflow-integration-postgres_postgres_1
3be46081e5e7 mongo:3 "docker-entrypoint.s…" 20 seconds ago Up 17 seconds (healthy) 27017/tcp airflow-integration-postgres_mongo_1
86de6cfc0749 apachepinot/pinot:latest "./bin/pinot-admin.s…" 20 seconds ago Restarting (0) Less than a second ago airflow-integration-postgres_pinot_1
b84549fa9971 cassandra:3.0 "docker-entrypoint.s…" 20 seconds ago Up 15 seconds (health: starting) 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp airflow-integration-postgres_cassandra_1
ef7b990c0eca redis:5.0.1 "docker-entrypoint.s…" 20 seconds ago Up 15 seconds (healthy) 0.0.0.0:26379->6379/tcp, :::26379->6379/tcp airflow-integration-postgres_redis_1
9879aad4d9b3 ghcr.io/apache/airflow-openldap:2.4.50-2021.07.04 "/container/tool/run…" 20 seconds ago Up 17 seconds (healthy) 389/tcp, 636/tcp airflow-integration-postgres_openldap_1
d0e3c6be6423 ghcr.io/apache/airflow-krb5-kdc-server:2021.07.04 "/entrypoint.sh /usr…" 20 seconds ago Up 17 seconds (healthy) airflow-integration-postgres_kdc-server-example-com_1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971535805
also trying t reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971502650
> It would be good to restore CI builds back to success!
Oh absolutely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#discussion_r752198383
##########
File path: scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh
##########
@@ -123,8 +124,10 @@ function run_airflow_testing_in_docker() {
"${DOCKER_COMPOSE_LOCAL[@]}" \
--project-name "airflow-${TEST_TYPE}-${BACKEND}" \
run airflow "${@}"
+ docker ps
Review comment:
Whoops! Putting this here meant that the exit_code of each test type was always 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971505217
> I am looking into it slowly...
I am glad there are poeple who also look at it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on a change in pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on a change in pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#discussion_r752335171
##########
File path: scripts/ci/testing/ci_run_single_airflow_test_in_docker.sh
##########
@@ -123,8 +124,10 @@ function run_airflow_testing_in_docker() {
"${DOCKER_COMPOSE_LOCAL[@]}" \
--project-name "airflow-${TEST_TYPE}-${BACKEND}" \
run airflow "${@}"
+ docker ps
Review comment:
Created this for the same: https://github.com/apache/airflow/pull/19682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971619590
Yeah. It's good to keep it !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971552644
I went ahead and added fixups to test it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971593458
Thanks for the investigation @khalidmammadov !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971496428
Thanks @potiuk. Although issue is confirmed to be with this openldap the build is still failing as you can see. I think there is some ordering or parallelism issue here that creating this openldap volume and failing the build. I am looking into it slowly... but if you know what might cause this please go ahead with fix. It would be good to restore CI builds back to success!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971549390
We should pin pinot to `apachepinot/pinot:0.8.0` and grafana to `grafana/grafana:8.2.4` just in case (those are the stable latest releases)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on pull request #19633: Fix failing CI phase with unhealthy container issue
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on pull request #19633:
URL: https://github.com/apache/airflow/pull/19633#issuecomment-971516470
Cant reproduce locally hence enabled debug to see if it's still openldap or some other container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org