You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "muzehyun (via GitHub)" <gi...@apache.org> on 2025/09/08 17:08:40 UTC
[D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
GitHub user muzehyun created a discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
Running airflow 3.0.6 with docker.
It works okay without docker.
Config: [airflow-config.txt](https://github.com/user-attachments/files/22215590/airflow-config.txt)
[docker-compose.yaml.txt](https://github.com/user-attachments/files/22215643/docker-compose.yaml.txt)
dag
```
import datetime
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
def foo(**kwargs):
return "Whatever you return gets printed in the logs"
with DAG(
dag_id="python_operator_dag",
start_date=datetime.datetime(2025, 1, 1),
schedule=None,
):
PythonOperator(task_id="foo", python_callable=foo)
```
Errors from worker log
```
-------------- celery@1df315592f1d v5.5.3 (immunity)
--- ***** -----
-- ******* ---- Linux-6.15.11-orbstack-00539-g9885ebd8e3f4-aarch64-with-glibc2.31 2025-09-08 16:55:52
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: airflow.providers.celery.executors.celery_executor:0xffff826b8100
- ** ---------- .> transport: redis://redis:6379/1
- ** ---------- .> results: postgresql://airflow:**@postgres:5432/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. execute_workload
2025-09-08 16:55:53.936632 [info ] Connected to redis://redis:6379/1 [celery.worker.consumer.connection]
2025-09-08 16:55:53.937856 [info ] mingle: searching for neighbors [celery.worker.consumer.mingle]
2025-09-08 16:55:54.946640 [info ] mingle: all alone [celery.worker.consumer.mingle]
2025-09-08 16:55:59.967371 [info ] celery@1df315592f1d ready. [celery.apps.worker]
2025-09-08 16:56:04.760294 [info ] Task execute_workload[6138f6e9-d8f5-4344-a81c-34a97e19c299] received [celery.worker.strategy]
2025-09-08 16:56:04.817455 [info ] [6138f6e9-d8f5-4344-a81c-34a97e19c299] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('01992a41-924a-70ba-8e75-91d168df4936'), task_id='foo', dag_id='python_operator_dag', run_id='manual__2025-09-08T16:56:04.394367+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('python_operator_dag.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=python_operator_dag/run_id=manual__2025-09-08T16:56:04.394367+00:00/task_id=foo/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-09-08 16:56:04.823034 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
2025-09-08 16:56:04.830088 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:05.837363 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:07.154363 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:08.187003 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2025-09-08 16:56:09.901427 [info ] Process exited [supervisor] exit_code=-9 pid=33 signal_sent=SIGKILL
2025-09-08 16:56:09.918525 [error ] Task execute_workload[6138f6e9-d8f5-4344-a81c-34a97e19c299] raised unexpected: ConnectError('[Errno 111] Connection refused') [celery.app.trace]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:101 in │
│ map_httpcore_exceptions │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:250 in handle_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py:256 in handle_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py:236 in handle_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:101 in handle_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:78 in handle_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py:124 in _connect │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_backends/sync.py:207 in connect_tcp │
│ │
│ /usr/local/lib/python3.10/contextlib.py:153 in __exit__ │
│ │
│ /usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py:14 in map_exceptions │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ConnectError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.10/site-packages/celery/app/trace.py:453 in trace_task │
│ │
│ /usr/local/lib/python3.10/site-packages/celery/app/trace.py:736 in __protected_call__ │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/providers/celery/executors/celery_executor_utils │
│ .py:171 in execute_workload │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:1829 in │
│ supervise │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:933 in start │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py:944 in │
│ _on_child_started │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/api/client.py:152 in start │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:1218 in patch │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:338 in wrapped_f │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:477 in __call__ │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:378 in iter │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:420 in exc_check │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:187 in reraise │
│ │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:451 in result │
│ │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:403 in __get_result │
│ │
│ /usr/local/lib/python3.10/site-packages/tenacity/__init__.py:480 in __call__ │
│ │
│ /usr/local/lib/python3.10/site-packages/airflow/sdk/api/client.py:735 in request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:825 in request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:914 in send │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:942 in _send_handling_auth │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:979 in _send_handling_redirects │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_client.py:1014 in _send_single_request │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:249 in handle_request │
│ │
│ /usr/local/lib/python3.10/contextlib.py:153 in __exit__ │
│ │
│ /usr/local/lib/python3.10/site-packages/httpx/_transports/default.py:118 in │
│ map_httpcore_exceptions │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ConnectError: [Errno 111] Connection refused
```
GitHub link: https://github.com/apache/airflow/discussions/55380
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
Posted by "jibrilhp (via GitHub)" <gi...@apache.org>.
GitHub user jibrilhp added a comment to the discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
Hi, I see you're having connection issues. I'm using a Kubernetes environment and solved my problem by changing the AIRFLOW__CORE__EXECUTION_API_SERVER_URL. I'm not sure if it applies to your exact case, but it might be worth checking if your worker can properly connect to the API server.
GitHub link: https://github.com/apache/airflow/discussions/55380#discussioncomment-14708124
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
Posted by "Tzinch21 (via GitHub)" <gi...@apache.org>.
GitHub user Tzinch21 deleted a comment on the discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
Faced with the same error.
Check that your environment variable refers to your api_server in docker
For example:
AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://api-server:8080/execution/
GitHub link: https://github.com/apache/airflow/discussions/55380#discussioncomment-14781889
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
Posted by "Tzinch21 (via GitHub)" <gi...@apache.org>.
GitHub user Tzinch21 added a comment to the discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
Faced with the same error.
Check that your environment variable refers to your api_server in docker
For example:
AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://api-server:8080/execution/
GitHub link: https://github.com/apache/airflow/discussions/55380#discussioncomment-14781889
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
You seem to mix webserver (airflow 2) and apiserver (airflow 3). your config and docker has webserver config, not api -server config - suggest you generate fresh config from airflow 3 and only apply changes you want to apply rather than copying config from airflow 2 - also you can look at the quick start docker-compose of ours (you will find it in a doc) for an inspiration for the api_server change in Airflow 3 and how to adapt your compose file.
Generally speaking you have **something** wrongly configured so that the worker cannot connect to the api-server (port or hostname)
GitHub link: https://github.com/apache/airflow/discussions/55380#discussioncomment-14343374
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker [airflow]
Posted by "Tzinch21 (via GitHub)" <gi...@apache.org>.
GitHub user Tzinch21 added a comment to the discussion: 'Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.' in Docker
Faced with the same error.
Check that your environment variable refers to your api_server in docker
For example:
AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://api-server:8080/execution/
GitHub link: https://github.com/apache/airflow/discussions/55380#discussioncomment-14781891
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org