You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcampos87 (via GitHub)" <gi...@apache.org> on 2025/04/29 13:33:18 UTC
[I] Webserver pod crashing when otel_on: True [airflow]
rcampos87 opened a new issue, #49957:
URL: https://github.com/apache/airflow/issues/49957
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have installed an otel collector running alongside my Airflow pods and now I'm trying to enable otel collector in aiflow but my webserver pod keeps crashing even though every other pod is working fine. When logging my webserver crash I get:
```
[2025-04-29T11:58:36.283+0000] {cli_parser.py:67} ERROR - Failed to load CLI commands from executor: CeleryKubernetesExecutor
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_parser.py", line 64, in <module>
executor, _ = ExecutorLoader.import_executor_cls(executor_name)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 302, in import_executor_cls
return _import_and_validate(executor_name.module_path), executor_name.connector_source
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 286, in _import_and_validate
executor = import_string(path)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/celery/executors/celery_kubernetes_executor.py", line 28, in <module>
from airflow.executors.base_executor import BaseExecutor
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/base_executor.py", line 36, in <module>
from airflow.stats import Stats
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/stats.py", line 78, in <module>
class Stats(metaclass=_Stats):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/stats.py", line 59, in __init__
from airflow.metrics import otel_logger
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/metrics/otel_logger.py", line 27, in <module>
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
File "/home/airflow/.local/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/http/metric_exporter/__init__.py", line 25, in <module>
from opentelemetry.exporter.otlp.proto.common._internal import (
File "/home/airflow/.local/lib/python3.10/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py", line 31, in <module>
from opentelemetry.proto.common.v1.common_pb2 import (
File "/home/airflow/.local/lib/python3.10/site-packages/opentelemetry/proto/common/v1/common_pb2.py", line 17, in <module>
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n*opentelemetry/proto/common/v1/common.proto\x12\x1dopentelemetry.proto.common.v1\"\x8c\x02\n\x08\x41nyValue\x12\x16\n\x0cstring_value\x18\x01 \x01(\tH\x00\x12\x14\n\nbool_value\x18\x02 \x01(\x08H\x00\x12\x13\n\tint_value\x18\x03 \x01(\x03H\x00\x12\x16\n\x0c\x64ouble_value\x18\x04 \x01(\x01H\x00\x12@\n\x0b\x61rray_value\x18\x05 \x01(\x0b\x32).opentelemetry.proto.common.v1.ArrayValueH\x00\x12\x43\n\x0ckvlist_value\x18\x06 \x01(\x0b\x32+.opentelemetry.proto.common.v1.KeyValueListH\x00\x12\x15\n\x0b\x62ytes_value\x18\x07 \x01(\x0cH\x00\x42\x07\n\x05value\"E\n\nArrayValue\x12\x37\n\x06values\x18\x01 \x03(\x0b\x32\'.opentelemetry.proto.common.v1.AnyValue\"G\n\x0cKeyValueList\x12\x37\n\x06values\x18\x01 \x03(\x0b\x32\'.opentelemetry.proto.common.v1.KeyValue\"O\n\x08KeyValue\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\x36\n\x05value\x18\x02 \x01(\x0b\x32\'.opentelemetry.proto.common.v1.AnyValue\"\x94\x01\n\x14InstrumentationScope\
x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0f\n\x07version\x18\x02 \x01(\t\x12;\n\nattributes\x18\x03 \x03(\x0b\x32\'.opentelemetry.proto.common.v1.KeyValue\x12 \n\x18\x64ropped_attributes_count\x18\x04 \x01(\rB{\n io.opentelemetry.proto.common.v1B\x0b\x43ommonProtoP\x01Z(go.opentelemetry.io/proto/otlp/common/v1\xaa\x02\x1dOpenTelemetry.Proto.Common.V1b\x06proto3')
TypeError: Couldn't build proto file into descriptor pool: duplicate file name opentelemetry/proto/common/v1/common.proto
```
I have checked and im able to see metrics being posted to otel collector, it is just the webserver pod crashing. If I try running `airflow webserver` inside the airflow worker pod, the server inits fine.
### What you think should happen instead?
_No response_
### How to reproduce
Installing otel collector with otel operator
Adding
```
config:
metrics:
otel_on: 'True'
otel_host: otel-collector
otel_port: 4318
otel_prefix: airflow
otel_interval_milliseconds: 30000
otel_ssl_active: False
```
Redeploying airflow with helm
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
```
config:
metrics:
otel_on: 'True'
otel_host: otel-collector
otel_port: 4318
otel_prefix: airflow
otel_interval_milliseconds: 30000
otel_ssl_active: False
```
### Anything else?
Trying to run `airflow webserver` inside airflow worker pod works and inits the gunicorn workers fine.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Webserver pod crashing when otel_on: True [airflow]
Posted by "rcampos87 (via GitHub)" <gi...@apache.org>.
rcampos87 commented on issue #49957:
URL: https://github.com/apache/airflow/issues/49957#issuecomment-2840895566
Upon deeper investigation this is due to a conflict with newrelic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Webserver pod crashing when otel_on: True [airflow]
Posted by "rcampos87 (via GitHub)" <gi...@apache.org>.
rcampos87 closed issue #49957: Webserver pod crashing when otel_on: True
URL: https://github.com/apache/airflow/issues/49957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org