You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "AutomationDev85 (via GitHub)" <gi...@apache.org> on 2026/05/18 13:53:13 UTC
[PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
AutomationDev85 opened a new pull request, #67115:
URL: https://github.com/apache/airflow/pull/67115
# Overview
Some Edge workers intermittently fail with the following error:
`2026-05-17T09:39:34.839980Z [error ] Task execution failed [airflow.providers.edge3.cli.worker] loc=worker.py:226
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/edge3/cli/worker.py", line 213, in _run_job_via_supervisor
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 2107, in supervise
exit_code = process.wait()
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1062, in wait
self._monitor_subprocess()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1127, in _monitor_subprocess
alive = self._service_subprocess(max_wait_time=max_wait_time) is None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 791, in _service_subprocess
events = self.selector.select(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed epoll object`
After analyzing the issue, we found that after the socket cleanup timeout fires and _cleanup_open_sockets() closes the selector, the monitor loop was not exited. On the next iteration it called selector.select() on the already-closed epoll object, causing the ValueError.
# Details of change:
- Add a break to exit the monitor loop immediately after the forced socket cleanup, preventing any further calls to selector.select() on the closed selector.
- Call _open_sockets.clear() inside _cleanup_open_sockets() to keep the socket registry consistent with the selector state after cleanup.
- Adapt unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #67115:
URL: https://github.com/apache/airflow/pull/67115#discussion_r3264689555
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1188,19 +1186,15 @@ def test_cleanup_sockets_after_delay(self, monkeypatch, mocker, time_machine):
proc._exit_code = 0
# Create a fake placeholder in the open socket weakref
- proc._open_sockets[mocker.MagicMock()] = "test placeholder"
- proc._process_exit_monotonic = time.time()
-
- mocker.patch.object(
- ActivitySubprocess,
- "_cleanup_open_sockets",
- side_effect=lambda: setattr(proc, "_open_sockets", {}),
- )
-
- time_machine.shift(2)
+ mock_socket = mocker.MagicMock()
Review Comment:
Maybe use `mocker.MagicMock(spec=socket.socket)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
Posted by "AutomationDev85 (via GitHub)" <gi...@apache.org>.
AutomationDev85 commented on code in PR #67115:
URL: https://github.com/apache/airflow/pull/67115#discussion_r3264732630
##########
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:
##########
@@ -1188,19 +1186,15 @@ def test_cleanup_sockets_after_delay(self, monkeypatch, mocker, time_machine):
proc._exit_code = 0
# Create a fake placeholder in the open socket weakref
- proc._open_sockets[mocker.MagicMock()] = "test placeholder"
- proc._process_exit_monotonic = time.time()
-
- mocker.patch.object(
- ActivitySubprocess,
- "_cleanup_open_sockets",
- side_effect=lambda: setattr(proc, "_open_sockets", {}),
- )
-
- time_machine.shift(2)
+ mock_socket = mocker.MagicMock()
Review Comment:
@amoghrajesh Thanks for this idea! I pushed the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 merged PR #67115:
URL: https://github.com/apache/airflow/pull/67115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #67115:
URL: https://github.com/apache/airflow/pull/67115#issuecomment-4486051593
Interesting. I sort of hoped that the stuck socket timeout never fired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ValueError when supervisor force-closes stuck sockets after timeout [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #67115:
URL: https://github.com/apache/airflow/pull/67115#issuecomment-4486653127
### Backport successfully created: v3-2-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/pull/67162"><img src="https://img.shields.io/badge/PR-67162-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org