You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/01/26 14:20:09 UTC
[I] TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail [airflow]
jason810496 opened a new issue, #61070:
URL: https://github.com/apache/airflow/issues/61070
## What
The test sometimes fails.
For example: https://github.com/apache/airflow/actions/runs/21358582533/job/61472786065?pr=60989
## Traceback Logs
summary:
```python
FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes - AssertionError: Span name 'task2' wasn't found in children span names. It's not a child of span 'otel_test_dag_with_pause_between_tasks'.
===================================================================== 1 failed, 7 passed, 10 skipped, 1 warning in 417.55s (0:06:57) =====================================================================
```
full traceback:
```python
Traceback (most recent call last):
File "/usr/python/bin/airflow", line 10, in <module>
sys.exit(main())
File "/opt/airflow/airflow-core/src/airflow/__main__.py", line 55, in main
args.func(args)
File "/opt/airflow/airflow-core/src/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
File "/opt/airflow/airflow-core/src/airflow/utils/cli.py", line 113, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/airflow-core/src/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
return func(*args, **kwargs)
File "/opt/airflow/airflow-core/src/airflow/cli/commands/api_server_command.py", line 120, in wrapper
return func(args)
File "/opt/airflow/airflow-core/src/airflow/cli/commands/api_server_command.py", line 164, in api_server
run_command_with_daemon_option(
File "/opt/airflow/airflow-core/src/airflow/cli/commands/daemon_utils.py", line 58, in run_command_with_daemon_option
check_if_pidfile_process_is_running(pid_file=pid, process_name=process_name)
File "/opt/airflow/airflow-core/src/airflow/utils/process_utils.py", line 370, in check_if_pidfile_process_is_running
raise AirflowException(f"The {process_name} is already running under PID {pid}.")
airflow.sdk.exceptions.AirflowException: The api_server is already running under PID 124.
2026-01-26T13:32:15.730154Z [error ] Exception while exporting metrics HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:545
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 541, in _receive_metrics
self._exporter.export(
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 203, in export
resp = self._export(serialized_data.SerializeToString())
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 173, in _export
return self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
/usr/python/lib/python3.10/site-packages/celery/platforms.py:841 SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
2026-01-26T13:32:17.299274Z [info ] Connected to redis://redis:6379/0 [celery.worker.consumer.connection] loc=connection.py:22
2026-01-26T13:32:17.305338Z [info ] mingle: searching for neighbors [celery.worker.consumer.mingle] loc=mingle.py:40
2026-01-26T13:32:18.316065Z [info ] mingle: all alone [celery.worker.consumer.mingle] loc=mingle.py:49
2026-01-26T13:32:18.330157Z [info ] celery@753b8cbd20cb ready. [celery.apps.worker] loc=worker.py:176
2026-01-26T13:32:19.434136Z [error ] Exception while exporting metrics HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)")) [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:545
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail [airflow]
Posted by "Abhishekmishra2808 (via GitHub)" <gi...@apache.org>.
Abhishekmishra2808 commented on issue #61070:
URL: https://github.com/apache/airflow/issues/61070#issuecomment-3818105209
HI, @jason810496
### Deep Dive: Root Cause Analysis
After reviewing the full traceback, the `AssertionError` is a secondary symptom. The primary cause is a **Networking/DNS failure** within the Breeze environment.
**Key Evidence from Traceback:**
* `socket.gaierror: [Errno -3] Temporary failure in name resolution`
* `urllib3.exceptions.NameResolutionError: Failed to resolve 'breeze-otel-collector'`
**What's happening:**
The OpenTelemetry SDK is attempting to export metrics/spans to `http://breeze-otel-collector:4318/v1/metrics`, but the hostname cannot be resolved. This leads to dropped spans, which is why `task2` is missing from the children spans during the test assertion.
Furthermore, there's a PID conflict (`api_server is already running under PID 124`), suggesting that the CI runner might be suffering from "leaked" processes from previous steps, which often interferes with Docker network stability.
### Proposed Fixes:
1. **Breeze Orchestration:** Ensure the `otel-collector` service is explicitly listed as a dependency for integration tests and has a healthy status check before the tests begin.
2. **Robustness:** Wrap the span assertion in a retry block (as previously suggested) to handle cases where the collector is briefly unreachable during startup.
3. **CI Cleanup:** Ensure a cleaner teardown of the `api_server` to prevent PID conflicts that might be locking network resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #61070: TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail
URL: https://github.com/apache/airflow/issues/61070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail [airflow]
Posted by "Abhishekmishra2808 (via GitHub)" <gi...@apache.org>.
Abhishekmishra2808 commented on issue #61070:
URL: https://github.com/apache/airflow/issues/61070#issuecomment-3822631039
@jason810496
I have investigated the logs and reproduced the flakiness. The `AssertionError` for `task2` was caused by a transient DNS resolution failure (`NameResolutionError`) where the Airflow components attempted to connect to the `breeze-otel-collector` before it was fully reachable on the Docker network.
I've submitted a PR that introduces a `wait_for_otel_collector` health check in the `setup_class` of the integration tests. This ensures the environment is stable and the collector is reachable before the test logic executes, preventing the race condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes sometimes fail [airflow]
Posted by "Abhishekmishra2808 (via GitHub)" <gi...@apache.org>.
Abhishekmishra2808 commented on issue #61070:
URL: https://github.com/apache/airflow/issues/61070#issuecomment-3818094701
Hi, @jason810496
### Analysis of the Flaky Failure
I've analyzed the logs from the recent CI failures for `test_scheduler_change_after_the_first_task_finishes`. The `AssertionError` regarding the missing `task2` span appears to be a race condition rather than a logic bug in the OTel integration itself.
**Observed Issues:**
1. **Collector Latency:** The integration tests rely on the `breeze-otel-collector`. Logs indicate occasional `NameResolutionError` or connection timeouts, suggesting the collector isn't always ready when the test begins its assertions.
2. **Eventual Consistency:** Since OTel spans are exported asynchronously, the test is asserting the existence of 'task2' before the collector has finished processing the trace from the second scheduler.
### Proposed Approach
I would like to work on a fix to make this test more robust. My proposed approach is:
* **Implement a Polling/Retry Mechanism:** Instead of a single assertion, I'll wrap the span validation in a retry loop (using a helper or `tenacity`) with a reasonable timeout (e.g., 10s). This allows for the inherent delay in telemetry propagation.
* **Environment Check:** Add a pre-test check to ensure the OTel collector endpoint is reachable before the test suite proceeds.
I'm happy to open a PR for this if the maintainers agree this is the right direction to reduce CI noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org