You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/03/03 07:31:22 UTC
[I] Flaky OTEL connection in integration tests [airflow]
jason810496 opened a new issue, #62769:
URL: https://github.com/apache/airflow/issues/62769
### Body
## What
The test sometimes fails.
For example: https://github.com/apache/airflow/actions/runs/22609497337/job/65509753048?pr=62758
## Traceback Logs
summary:
```python
======================================================================================== short test summary info =========================================================================================
FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_change_after_the_first_task_finishes - AssertionError: Span name 'task2' wasn't found in children span names. It's not a child of span 'otel_test_dag_with_pause_between_tasks'.
FAILED airflow-core/tests/integration/otel/test_otel.py::TestOtelIntegration::test_scheduler_exits_gracefully_in_the_middle_of_the_first_task - AssertionError: Span name 'task1_sub_span3' wasn't found in children span names. It's not a child of span 'task1_sub_span2'.
===================================================================== 2 failed, 7 passed, 10 skipped, 1 warning in 580.16s (0:09:40) =====================================================================
```
<details>
<summary>full traceback:</summary>
```python
Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
self._exporter.export(
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
resp = self._export(serialized_data, deadline_sec - time())
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Warning: ] Invalid type UUID for attribute 'airflow.task.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types [opentelemetry.attributes] loc=__init__.py:111
Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
self._exporter.export(
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
resp = self._export(serialized_data, deadline_sec - time())
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
self._exporter.export(
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
resp = self._export(serialized_data, deadline_sec - time())
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
Error: ] Exception while exporting metrics [opentelemetry.sdk.metrics._internal.export] loc=__init__.py:552
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 205, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 204, in _new_conn
sock = connection.create_connection(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/python/lib/python3.10/socket.py", line 967, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 500, in request
self.endheaders()
File "/usr/python/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/python/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 331, in connect
self.sock = self._new_conn()
File "/usr/python/lib/python3.10/site-packages/urllib3/connection.py", line 211, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 644, in send
resp = conn.urlopen(
File "/usr/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/usr/python/lib/python3.10/site-packages/urllib3/util/retry.py", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 548, in _receive_metrics
self._exporter.export(
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 234, in export
resp = self._export(serialized_data, deadline_sec - time())
File "/usr/python/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 213, in _export
resp = self._session.post(
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/python/lib/python3.10/site-packages/requests/adapters.py", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='breeze-otel-collector', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NameResolutionError("HTTPConnection(host='breeze-otel-collector', port=4318): Failed to resolve 'breeze-otel-collector' ([Errno -3] Temporary failure in name resolution)"))
-- err-end
ERROR integration.otel.otel_utils:otel_utils.py:146 Failed to parse JSON: {
"resource_metrics": [
{
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.39.1",
"service.name": "airflow"
},
/root/airflow/logs/dag_id=otel_test_dag_with_pause_in_task/run_id=manual__2026-03-03T05:53:30.782643+00:00/task_id=task1/attempt=1.log
File "/usr/python/lib/python3.10/logging/__init__.py", line 1100, in emit
msg = self.format(record)
File "/usr/python/lib/python3.10/logging/__init__.py", line 943, in format
return fmt.format(record)
File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 1154, in format
ed = cast(EventDict, proc(logger, meth_name, ed))
File "/usr/python/lib/python3.10/site-packages/structlog/stdlib.py", line 764, in __call__
if len(args) == 1 and isinstance(args[0], dict) and args[0]:
File "/usr/python/bin/celery", line 10, in <module>
sys.exit(main())
File "/usr/python/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/usr/python/lib/python3.10/site-packages/celery/bin/celery.py", line 227, in main
return celery(auto_envvar_prefix="CELERY")
File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
return self.main(*args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/python/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/python/lib/python3.10/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/click/decorators.py", line 34, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/celery/bin/base.py", line 135, in caller
return f(ctx, *args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/celery/bin/worker.py", line 367, in worker
worker.start()
File "/usr/python/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/usr/python/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/usr/python/lib/python3.10/site-packages/celery/concurrency/base.py", line 130, in start
self.on_start()
File "/usr/python/lib/python3.10/site-packages/celery/concurrency/prefork.py", line 112, in on_start
P = self._pool = Pool(processes=self.limit,
File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 475, in __init__
super().__init__(processes, *args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1046, in __init__
self._create_worker_process(i)
File "/usr/python/lib/python3.10/site-packages/celery/concurrency/asynpool.py", line 493, in _create_worker_process
return super()._create_worker_process(i)
File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 1158, in _create_worker_process
w.start()
File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 120, in start
self._popen = self._Popen(self)
File "/usr/python/lib/python3.10/site-packages/billiard/context.py", line 331, in _Popen
return Popen(process_obj)
File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 22, in __init__
self._launch(process_obj)
File "/usr/python/lib/python3.10/site-packages/billiard/popen_fork.py", line 77, in _launch
code = process_obj._bootstrap()
File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 323, in _bootstrap
self.run()
File "/usr/python/lib/python3.10/site-packages/billiard/process.py", line 110, in run
self._target(*self._args, **self._kwargs)
File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 292, in __call__
sys.exit(self.workloop(pid=pid))
File "/usr/python/lib/python3.10/site-packages/billiard/pool.py", line 362, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 694, in fast_trace_task
R, I, T, Rstr = tasks[task].__trace__(
File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 479, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/python/lib/python3.10/site-packages/celery/app/trace.py", line 779, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/airflow/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
supervise(
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 2090, in supervise
process = ActivitySubprocess.start(
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 1004, in start
proc: Self = super().start(id=what.id, client=client, target=target, logger=logger, **kwargs)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 514, in start
_fork_main(child_requests, child_stdout, child_stderr, child_logs.fileno(), target)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 413, in _fork_main
target()
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main
main()
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1825, in main
state, _, error = run(ti, context, log)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1236, in run
result = _execute_task(context=context, ti=ti, log=log)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1651, in _execute_task
result = ctx.run(execute, context=context)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper
return func(self, *args, **kwargs)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/decorator.py", line 360, in execute
return_value = super().execute(context)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 443, in wrapper
return func(self, *args, **kwargs)
File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 232, in execute
return_value = self.execute_callable()
File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py", line 255, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/callback_runner.py", line 97, in run
return func(*args, **kwargs)
File "/opt/airflow/airflow-core/tests/integration/otel/dags/otel_test_dag_with_pause_in_task.py", line 76, in task1
logger.info("Found ti.context_carrier: %s.", context_carrier)
{
"name": "task1_sub_span3",
"context": {
"trace_id": "0x186713f4ffcd23351586d08d9fd5ed34",
"span_id": "0x57ffb0578ea05d12",
"trace_state": "[]"
},
"kind": "SpanKind.INTERNAL",
"parent_id": "0x70dad54d80c553f6",
"start_time": "2026-03-03T05:53:42.271939Z",
"end_time": "2026-03-03T05:53:42.272071Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"attr3": "val3"
},
"events": [],
"links": [],
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.39.1",
"service.name": "airflow"
},
"schema_url": ""
}
}
```
</details>
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky OTEL connection in integration tests [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 closed issue #62769: Flaky OTEL connection in integration tests
URL: https://github.com/apache/airflow/issues/62769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky OTEL connection in integration tests [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #62769:
URL: https://github.com/apache/airflow/issues/62769#issuecomment-4470535009
Close as https://github.com/apache/airflow/issues/62769#issuecomment-4041123551 metioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky OTEL connection in integration tests [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #62769:
URL: https://github.com/apache/airflow/issues/62769#issuecomment-3991046651
hey @jason810496 is this issue happening because the OTEL exporter isn’t ready to export all the tasks?
I wanted to try working on it, but I’m a bit confused about how this whole OTEL + CI setup works.
Is there a chance I can contribute to this and learn while working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky OTEL connection in integration tests [airflow]
Posted by "andreahlert (via GitHub)" <gi...@apache.org>.
andreahlert commented on issue #62769:
URL: https://github.com/apache/airflow/issues/62769#issuecomment-4041123551
@jason810496
I believe that this was resolved by two PRs that landed in main:
- #61642 added wait_for_otel_collector which polls the collector via TCP before running tests, fixing the DNS resolution flakiness that caused the failures.
- #56150 rewrote the OTEL integration tests to use standard OTel environment variables, which also touched the test structure.
Is that okay? If yes, can you close the issue please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org