You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2023/02/19 20:26:06 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #29616: Refactor docker-compose quick start test
Taragolis opened a new pull request, #29616:
URL: https://github.com/apache/airflow/pull/29616
There is quite a few tests failed with the same error during the PR CI pipeline
```console
> assert dag_state == "success"
E AssertionError: assert 'queued' == 'success'
E - success
E + queued
```
According to the logs, services started successfully, however no logs from scheduler since it started.
I've didn't see that this happen in `main`, so I assume it only happen during run on public runners.
Add minor refactoring of this test:
- Set environment variables and current working directory by `pytest`
- Print additional information during failure: Docker and docker Compose versions, all DagRuns for test DAG, DAG Detail
- Do not copy `example_bash_operator` DAG because we set `AIRFLOW__CORE__LOAD_EXAMPLES` to `true` and always use bundled one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed pull request #29616: Refactor docker-compose quick start test
URL: https://github.com/apache/airflow/pull/29616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437905137
Closed/Reopened to re-run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437330747
Yeah as soon as just run a go to do my daily routine it finally failed š„ šÆ š¢
That is quite a bit interesting thing, some of them mostly could be a "wrong assumptions"
Dag Runs
---
```console
HTTP: GET dags/example_bash_operator/dagRuns
{'dag_runs': [{'conf': {},
'dag_id': 'example_bash_operator',
'dag_run_id': 'test_dag_run_id',
'data_interval_end': '2023-02-20T00:00:00+00:00',
'data_interval_start': '2023-02-19T00:00:00+00:00',
'end_date': None,
'execution_date': '2023-02-20T10:30:00.702880+00:00',
'external_trigger': True,
'last_scheduling_decision': None,
'logical_date': '2023-02-20T10:30:00.702880+00:00',
'note': None,
'run_type': 'manual',
'start_date': None,
'state': 'queued'}],
'total_entries': 1}
```
`example_bash_operator` DAG has scheduling interval, as result we should see here 2 DAG Runs, first for scheduled and second manual, in this case we could see only one - manual which created during the test.
Scheduler Logs
---
```console
airflow-scheduler_1 |
airflow-scheduler_1 | BACKEND=redis
airflow-scheduler_1 | DB_HOST=redis
airflow-scheduler_1 | DB_PORT=6379
airflow-scheduler_1 |
airflow-scheduler_1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow-scheduler_1 | ____________ _____________
airflow-scheduler_1 | ____ |__( )_________ __/__ /________ __
airflow-scheduler_1 | ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
airflow-scheduler_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
airflow-scheduler_1 | _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
airflow-scheduler_1 | [2023-02-20T10:29:14.618+0000] {executor_loader.py:114} INFO - Loaded executor: CeleryExecutor
airflow-scheduler_1 | [2023-02-20T10:29:14.664+0000] {scheduler_job.py:724} INFO - Starting the scheduler
airflow-scheduler_1 | [2023-02-20T10:29:14.665+0000] {scheduler_job.py:731} INFO - Processing each file at most -1 times
airflow-scheduler_1 | [2023-02-20T10:29:14.669+0000] {manager.py:164} INFO - Launched DagFileProcessorManager with pid: 33
airflow-scheduler_1 | [2023-02-20T10:29:14.671+0000] {scheduler_job.py:1437} INFO - Resetting orphaned tasks for active dag runs
airflow-scheduler_1 | [2023-02-20T10:29:14.685+0000] {settings.py:61} INFO - Configured default timezone Timezone('UTC')
```
Thats all, seems like it scheduler is just hang but service reported that it healthy. Is it problem with recent changes in health check https://github.com/apache/airflow/pull/29408 and maybe problem with simple http server in scheduler.
I would add output from `/health` endpoint in case of failure
Docker services after test failure
---
```console
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8da8ebd97f17 ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c "/usr/bin/dumb-init ā¦" 2 minutes ago Up 2 minutes (healthy) 8080/tcp quick-start_airflow-triggerer_1
88a829428ce8 ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c "/usr/bin/dumb-init ā¦" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp quick-start_airflow-webserver_1
f3baa9496225 ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c "/usr/bin/dumb-init ā¦" 2 minutes ago Up 2 minutes (healthy) 8080/tcp quick-start_airflow-scheduler_1
134b3356ed96 ghcr.io/apache/airflow/main/prod/python3.7:a8723aa63be724652809c141714af95493aea68c "/usr/bin/dumb-init ā¦" 2 minutes ago Up 2 minutes (healthy) 8080/tcp quick-start_airflow-worker_1
a5f5e8250820 redis:latest "docker-entrypoint.sā¦" 3 minutes ago Up 3 minutes (healthy) 6379/tcp quick-start_redis_1
de963f245166 postgres:13 "docker-entrypoint.sā¦" 3 minutes ago Up 3 minutes (healthy) 5432/tcp quick-start_postgres_1
```
All healthy, that mean initially services pass health check after start time
Versions
---
```console
$ docker version
Client:
Version: 20.10.23+azure-2
API version: 1.41
Go version: go1.19.6
Git commit: 715524332ff91d0f9ec5ab2ec95f051456ed1dba
Built: Wed Jan 18 20:42:16 UTC 2023
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.22+azure-1
API version: 1.41 (minimum version 1.12)
Go version: go1.18.9
Git commit: 42c8b314993e5eb3cc2776da0bbe41d5eb4b707b
Built: Thu Dec 15 22:17:04 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.18+azure-1
GitCommit: 2456e983eb9e37e47538f59ea18f2043c9a73640
runc:
Version: 1.1.4
GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925
docker-init:
Version: 0.19.0
GitCommit:
```
```console
$ docker-compose version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
```
That is more interesting. I've seen before that statics checks sometimes failed with particular this version of docker `20.10.23+azure-2` and didn't seen that this happen in docker without `azure-X`.
Another strange things
---
`Prepare Breeze and PROD image` step have a lot of errors witch refers to permission denied
```console
Received 27910740 of 32105044 (86.9%), 26.6 MBs/sec
Received 32105044 of 32105044 (100.0%), 29.8 MBs/sec
Cache Size: ~31 MB (32105044 B)
/usr/bin/tar -xf /home/runner/work/_temp/00fdf96d-139b-4[95](https://github.com/apache/airflow/actions/runs/4222264319/jobs/7330883288#step:4:100)4-ad8c-852b0f051104/cache.tgz -P -C /home/runner/work/airflow/airflow -z
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared/lib: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared/lib/python3.7: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared/lib/python3.7/site-packages: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared/lib/python3.7/site-packages/_distutils_hack: Cannot mkdir: No such file or directory
/usr/bin/tar: ../../../../.local: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.local/pipx/shared/lib/python3.7/site-packages/_distutils_hack/override.py: Cannot open: No such file or directory
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437277562
> The more you look on error the less chance that this error will reproduce roll_eyes rofl
Heisentest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437988917
Something new š®
```console
airflow-scheduler_1 |
airflow-scheduler_1 | BACKEND=redis
airflow-scheduler_1 | DB_HOST=redis
airflow-scheduler_1 | DB_PORT=6379
airflow-scheduler_1 |
airflow-scheduler_1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow-scheduler_1 | ____________ _____________
airflow-scheduler_1 | ____ |__( )_________ __/__ /________ __
airflow-scheduler_1 | ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
airflow-scheduler_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
airflow-scheduler_1 | _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
airflow-scheduler_1 | [2023-02-21T07:28:42.954+0000] {executor_loader.py:114} INFO - Loaded executor: CeleryExecutor
airflow-scheduler_1 | [2023-02-21T07:28:43.045+0000] {scheduler_job.py:724} INFO - Starting the scheduler
airflow-scheduler_1 | [2023-02-21T07:28:43.054+0000] {scheduler_job.py:731} INFO - Processing each file at most -1 times
airflow-scheduler_1 | [2023-02-21T07:28:43.061+0000] {manager.py:164} INFO - Launched DagFileProcessorManager with pid: 32
airflow-scheduler_1 | [2023-02-21T07:28:43.071+0000] {scheduler_job.py:1437} INFO - Resetting orphaned tasks for active dag runs
airflow-scheduler_1 | [2023-02-21T07:28:43.080+0000] {settings.py:61} INFO - Configured default timezone Timezone('UTC')
airflow-scheduler_1 | [2023-02-21T07:28:46.099+0000] {scheduler_job.py:788} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
airflow-scheduler_1 | Traceback (most recent call last):
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 771, in _execute
airflow-scheduler_1 | self._run_scheduler_loop()
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 899, in _run_scheduler_loop
airflow-scheduler_1 | num_queued_tis = self._do_scheduling(session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1006, in _do_scheduling
airflow-scheduler_1 | num_queued_tis = self._critical_section_enqueue_task_instances(session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 589, in _critical_section_enqueue_task_instances
airflow-scheduler_1 | queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 280, in _executable_task_instances_to_queued
airflow-scheduler_1 | pools = Pool.slots_stats(lock_rows=True, session=session)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 73, in wrapper
airflow-scheduler_1 | return func(*args, **kwargs)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/pool.py", line 174, in slots_stats
airflow-scheduler_1 | .group_by(TaskInstance.pool, TaskInstance.state)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2773, in all
airflow-scheduler_1 | return self._iter().all()
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1129, in all
airflow-scheduler_1 | return self._allrows()
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows
airflow-scheduler_1 | rows = self._fetchall_impl()
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl
airflow-scheduler_1 | return list(self.iterator)
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
airflow-scheduler_1 | fetch = cursor._raw_all_rows()
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 393, in _raw_all_rows
airflow-scheduler_1 | return [make_row(row) for row in rows]
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 393, in <listcomp>
airflow-scheduler_1 | return [make_row(row) for row in rows]
airflow-scheduler_1 | RuntimeError: number of values in row (1) differ from number of column processors (3)
airflow-scheduler_1 | [2023-02-21T07:28:47.137+0000] {process_utils.py:133} INFO - Sending Signals.SIGTERM to group 32. PIDs of all processes in the group: [44, 45, 32]
airflow-scheduler_1 | [2023-02-21T07:28:47.138+0000] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 32
airflow-scheduler_1 | [2023-02-21T07:28:49.462+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=44, status='terminated', started='07:28:45') (44) terminated with exit code None
airflow-scheduler_1 | [2023-02-21T07:28:49.471+0000] {process_utils.py:256} INFO - Waiting up to 5 seconds for processes to exit...
airflow-scheduler_1 | [2023-02-21T07:28:49.482+0000] {process_utils.py:256} INFO - Waiting up to 5 seconds for processes to exit...
airflow-scheduler_1 | [2023-02-21T07:28:49.494+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=45, status='terminated', started='07:28:46') (45) terminated with exit code None
airflow-scheduler_1 | [2023-02-21T07:28:49.500+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=32, status='terminated', exitcode=0, started='07:28:42') (32) terminated with exit code 0
airflow-scheduler_1 | [2023-02-21T07:28:49.500+0000] {scheduler_job.py:[800](https://github.com/apache/airflow/actions/runs/4229748945/jobs/7347366817#step:5:802)} INFO - Exited execute loop
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1444510331
Well, I guess we if we could add only this changes which might help to get a bit more information rather than previously in case if the error become "popular" again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1436664937
The more you look on error the less chance that this error will reproduce š š¤£
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #29616:
URL: https://github.com/apache/airflow/pull/29616#discussion_r1112728077
##########
docker_tests/test_docker_compose_quick_start.py:
##########
@@ -114,53 +104,60 @@ def wait_for_terminal_dag_state(dag_id, dag_run_id):
break
-def test_trigger_dag_and_wait_for_result():
+def test_trigger_dag_and_wait_for_result(tmp_path_factory, monkeypatch):
+ """Simple test which reproduce setup docker-compose environment and trigger example dag."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-quick-start")
+ monkeypatch.chdir(tmp_dir)
+ monkeypatch.setenv("AIRFLOW_IMAGE_NAME", docker_image)
+ monkeypatch.setenv("COMPOSE_PROJECT_NAME", "quick-start")
+
compose_file_path = (
SOURCE_ROOT / "docs" / "apache-airflow" / "howto" / "docker-compose" / "docker-compose.yaml"
)
+ copyfile(compose_file_path, tmp_dir / "docker-compose.yaml")
- with tempfile.TemporaryDirectory() as tmp_dir, tmp_chdir(tmp_dir), mock.patch.dict(
- "os.environ", AIRFLOW_IMAGE_NAME=docker_image
- ):
- copyfile(str(compose_file_path), f"{tmp_dir}/docker-compose.yaml")
- os.mkdir(f"{tmp_dir}/dags")
- os.mkdir(f"{tmp_dir}/logs")
- os.mkdir(f"{tmp_dir}/plugins")
- (Path(tmp_dir) / ".env").write_text(f"AIRFLOW_UID={subprocess.check_output(['id', '-u']).decode()}\n")
- print(".emv=", (Path(tmp_dir) / ".env").read_text())
- copyfile(
- str(SOURCE_ROOT / "airflow" / "example_dags" / "example_bash_operator.py"),
- f"{tmp_dir}/dags/example_bash_operator.py",
- )
+ # Create required directories for docker compose quick start howto
+ for subdir in ("dags", "logs", "plugins"):
+ (tmp_dir / subdir).mkdir()
+
+ dot_env_file = Path(tmp_dir) / ".env"
Review Comment:
No idea why I decide wrap `pathlib.Path` object into `pathlib.Path` š¤£
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1442289253
Yeah. I noticed it too :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437911170
> That is more interesting. I've seen before that statics checks sometimes failed with particular this version of docker 20.10.23+azure-2 and didn't seen that this happen in docker without azure-X.
Those tests seems to only fail on public runners (that's where azure-* is added in version) - so I think this is just coincidence.
> Thats all, seems like it scheduler is just hang but service reported that it healthy. Is this a problem with recent changes in health check https://github.com/apache/airflow/pull/29408 or maybe problem with simple http server in scheduler?
I agree. I think the problem was triggered by #29408 - but It would be great to get to the bottom of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1441985039
`Tests / Test docker-compose quick start (pull_request)` just stop failing and potentially we might fix it by another PR or it became rare in public runners :octocat:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis merged pull request #29616: Refactor docker-compose quick start test
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis merged PR #29616:
URL: https://github.com/apache/airflow/pull/29616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #29616: Refactor docker-compose quick start test
URL: https://github.com/apache/airflow/pull/29616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #29616: Refactor docker-compose quick start test
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #29616:
URL: https://github.com/apache/airflow/pull/29616#discussion_r1112580793
##########
docker_tests/test_docker_compose_quick_start.py:
##########
@@ -114,53 +104,60 @@ def wait_for_terminal_dag_state(dag_id, dag_run_id):
break
-def test_trigger_dag_and_wait_for_result():
+def test_trigger_dag_and_wait_for_result(tmp_path_factory, monkeypatch):
+ """Simple test which reproduce setup docker-compose environment and trigger example dag."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-quick-start")
+ monkeypatch.chdir(tmp_dir)
+ monkeypatch.setenv("AIRFLOW_IMAGE_NAME", docker_image)
+ monkeypatch.setenv("COMPOSE_PROJECT_NAME", "quick-start")
+
compose_file_path = (
SOURCE_ROOT / "docs" / "apache-airflow" / "howto" / "docker-compose" / "docker-compose.yaml"
)
+ copyfile(compose_file_path, tmp_dir / "docker-compose.yaml")
- with tempfile.TemporaryDirectory() as tmp_dir, tmp_chdir(tmp_dir), mock.patch.dict(
- "os.environ", AIRFLOW_IMAGE_NAME=docker_image
- ):
- copyfile(str(compose_file_path), f"{tmp_dir}/docker-compose.yaml")
- os.mkdir(f"{tmp_dir}/dags")
- os.mkdir(f"{tmp_dir}/logs")
- os.mkdir(f"{tmp_dir}/plugins")
- (Path(tmp_dir) / ".env").write_text(f"AIRFLOW_UID={subprocess.check_output(['id', '-u']).decode()}\n")
- print(".emv=", (Path(tmp_dir) / ".env").read_text())
- copyfile(
- str(SOURCE_ROOT / "airflow" / "example_dags" / "example_bash_operator.py"),
- f"{tmp_dir}/dags/example_bash_operator.py",
- )
+ # Create required directories for docker compose quick start howto
+ for subdir in ("dags", "logs", "plugins"):
+ (tmp_dir / subdir).mkdir()
+
+ dot_env_file = Path(tmp_dir) / ".env"
Review Comment:
```suggestion
dot_env_file = Path(tmp_dir, ".env")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1437294393
BTW. This one failed. And the problem is that `wait_for_terminal_state" simply exits after 80 tries when the task is "queued" . But why ? Hard to say. I have a feeling we have a race condition somewhere in scheduler (and likely it's a "real" one affecting tasks converted to "queued" right in the middle of some scheduler processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29616: Refactor docker-compose quick start test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29616:
URL: https://github.com/apache/airflow/pull/29616#issuecomment-1444564858
Yes. We should
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org