You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kolfild26 (via GitHub)" <gi...@apache.org> on 2025/05/04 02:07:09 UTC
[I] Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response [airflow]
kolfild26 opened a new issue, #50158:
URL: https://github.com/apache/airflow/issues/50158
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Airflow 3.0.0 installed via [docker-compose.yaml](https://airflow.apache.org/docs/apache-airflow/3.0.0/docker-compose.yaml). All services are healthy. When I trigger a dagrun, its tasks get stuck in a queued state infinitely.
Scheduler log:
<details>
```
{scheduler_job_runner.py:925} ERROR - DAG 'jirabot_asm' for task instance <TaskInstance: jirabot_asm.is_working manual__2025-05-04T01:18:52.104990+00:00 [queued]> not found in serialized_dag table
{scheduler_job_runner.py:941} ERROR - Executor CeleryExecutor(parallelism=64) reported that the task instance <TaskInstance: jirabot_asm.is_working manual__2025-05-04T01:18:52.104990+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
```
</details>
the worker log has a bit more :
<details>
```
request.url = http://airflow-apiserver:8080/execution/task-instances/01969804-a504-78d3-8383-b5ff3044956d/run
[2025-05-03T21:19:26.000+0000] {_trace.py:47} DEBUG - connect_tcp.started host='airflow-apiserver' port=8080 local_address=None timeout=5.0 socket_options=None
[2025-05-03T21:19:26.001+0000] {_trace.py:47} DEBUG - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7fc0b326b7d0>
[2025-05-03T21:19:26.001+0000] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PATCH']>
[2025-05-03T21:19:26.002+0000] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-03T21:19:26.002+0000] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PATCH']>
[2025-05-03T21:19:26.002+0000] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-03T21:19:26.003+0000] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PATCH']>
[2025-05-03T21:19:26.003+0000] {_trace.py:47} DEBUG - receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
[2025-05-03T21:19:26.003+0000] {_trace.py:47} DEBUG - response_closed.started
[2025-05-03T21:19:26.003+0000] {_trace.py:47} DEBUG - response_closed.complete
2025-05-04 00:19:26 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=31 signal_sent=SIGKILL
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
yield
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
return self._connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
) = self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
event = self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 231, in _receive_event
raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.
```
</details>
At first I assumed it might be connected with Celery, but LocalExecutor behaves the same way. Next, instead of a docker compose I installed via the python venv and... dags runs successfully in there.
But I've no idea what causes the [RemoteProtocolError](https://github.com/encode/httpcore/blob/master/httpcore/_sync/http11.py#L231) in case of a docker compose deployment. I even didn't make any changes in a yaml file.
Might this be a bug or I still missed any stuff? Any help would be appreciated.
### What you think should happen instead?
_No response_
### How to reproduce
I am using the official [docker-compose.yaml](https://airflow.apache.org/docs/apache-airflow/3.0.0/docker-compose.yaml) with no significant customization.
docker compose version v2.27.1
### Operating System
OEL 9.4
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response [airflow]
Posted by "emredjan (via GitHub)" <gi...@apache.org>.
emredjan commented on issue #50158:
URL: https://github.com/apache/airflow/issues/50158#issuecomment-3052233176
I am having a similar issue, but with a virtualenv deployment. Both LocalExecutor and CeleryExecutor fails with `RemoteProtocolError: Server disconnected without sending a response.` messages after upgrading to Airflow 3.0.2. Everything was working fine on 2.x. I updated the configuration with the new defaults, but nothing I changed made a difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #50158:
URL: https://github.com/apache/airflow/issues/50158#issuecomment-3052934589
Likely resource use possibly you need more memory or other resources. Apparently something killed your process with -9 - which usually means that. But you neeed to look in detail in your deployment - the **something** that killed the process forecully most likely printed why it did so - but it's totall up to your deployment and your look in your deployment why and how to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #50158: Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response
URL: https://github.com/apache/airflow/issues/50158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Worker doesn't execute tasks. RemoteProtocolError - Server disconnected without sending a response [airflow]
Posted by "kolfild26 (via GitHub)" <gi...@apache.org>.
kolfild26 commented on issue #50158:
URL: https://github.com/apache/airflow/issues/50158#issuecomment-3203014998
For those who read this and face the same problem:
- Check [the docs for 3.* regarding certificates](https://airflow.apache.org/docs/apache-airflow/stable/howto/run-with-self-signed-certificate.html#generating-the-certificate)
- Look at these threads ([53493](https://github.com/apache/airflow/issues/53493), [50726](https://github.com/apache/airflow/discussions/50726) and [slack](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1755543330505439?thread_ts=1755542914.188539&cid=CCQ7EGB1P)), they are might be helpfull.
In my case, this was caused by an incorrect SSL certificate (not satisfying the new requirements above), not by a lack of resources.
Unfortunately, there is no (?) proper verification for this, as RemoteProtocolError (thrown by the Airflow SDK itself) doesn't reveal much about the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org