You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "MichaelRBlack (via GitHub)" <gi...@apache.org> on 2026/03/20 00:31:51 UTC
[PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
MichaelRBlack opened a new pull request, #63959:
URL: https://github.com/apache/airflow/pull/63959
## Backport of #44346 to v2-11-stable
This is a backport of #44346 (merged to `main` Nov 2024) to the v2-11 maintenance branch. The fix was never cherry-picked to v2.
### Problem
`OTLPMetricExporter` and `OTLPSpanExporter` in `otel_logger.py` and `otel_tracer.py` have a hardcoded `headers={"Content-Type": "application/json"}` parameter. These exporters serialize data as **protobuf** and automatically set `Content-Type: application/x-protobuf`. The hardcoded override tells the OpenTelemetry Collector to decode the payload as JSON, but the bytes are protobuf — causing **100% export failure**:
```
Failed to export metrics batch code: 500,
reason: {"code": 13, "message": "failed to marshal error message"}
```
This means OTEL metrics and traces are **completely broken** for every Airflow 2.x user sending to a standard OTEL Collector.
As a secondary issue, the hardcoded `headers` parameter also prevents users from configuring custom headers via the standard `OTEL_EXPORTER_OTLP_HEADERS` environment variable (e.g., for authentication with hosted backends like Grafana Cloud or Logfire).
### Fix
Remove the `headers={"Content-Type": "application/json"}` parameter from both `OTLPMetricExporter` and `OTLPSpanExporter`, allowing the SDK to use its correct default (`application/x-protobuf`).
### Testing
Verified on a production Airflow 2.11.1 cluster sending to an OpenTelemetry Collector → Mimir pipeline:
- **Before fix**: every 30s export batch fails with HTTP 500 `"failed to marshal error message"`
- **After fix**: zero export errors, metrics immediately visible in Mimir
```python
# Reproducer — run from an Airflow pod:
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
good = OTLPMetricExporter(endpoint=endpoint)
bad = OTLPMetricExporter(endpoint=endpoint, headers={"Content-Type": "application/json"})
good.export(metrics_data) # SUCCESS
bad.export(metrics_data) # FAILURE — 500 "failed to marshal error message"
```
### Justification for v2 backport
This is a critical bug fix — OTEL metrics and traces are entirely non-functional in every Airflow 2.x release. With Airflow 2.x EOL approaching (April 2026), this fix would allow the remaining v2 user base to use OTEL monitoring for the remainder of the support window.
^ This diffance is:
- `airflow/metrics/otel_logger.py`: 1 line removed
- `airflow/traces/otel_tracer.py`: 1 line changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63959:
URL: https://github.com/apache/airflow/pull/63959#issuecomment-4529930716
@MichaelRBlack This draft PR has been inactive for 13 days since the last triage comment and no response from the author. Closing to keep the queue clean.
You are welcome to reopen this PR when you resume work, or to open a new one addressing the issues previously raised. There is no rush — take your time.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
---
Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63959:
URL: https://github.com/apache/airflow/pull/63959#issuecomment-4097436489
@MichaelRBlack This PR has been converted to **draft** because it does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Build docs**: Failing: CI image checks / Build documentation (--docs-only). Run `breeze build-docs` locally to reproduce. See [Build docs docs](https://github.com/apache/airflow/blob/main/contributing-docs/16_documentation.rst).
- :x: **Provider tests**: Failing: Lowest direct dependency providers tests / All-core:LowestDeps-Postgres:13:3.10. Run provider tests with `breeze run pytest <provider-test-path> -xvs`. See [Provider tests docs](https://github.com/apache/airflow/blob/main/contributing-docs/12_provider_distributions.rst).
- :x: **Other failing CI checks**: Failing: CI image checks / Build documentation (--spellcheck-only). Run `prek run --from-ref main` locally to reproduce. See [static checks docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
**What to do next:**
- The comment informs you what you need to do.
- Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
- There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
- Maintainers will then proceed with a normal review.
Converting a PR to draft is **not** a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #63959:
URL: https://github.com/apache/airflow/pull/63959#discussion_r3025336685
##########
airflow/traces/otel_tracer.py:
##########
@@ -269,7 +269,7 @@ def get_otel_tracer(cls) -> OtelTrace:
endpoint = f"{protocol}://{host}:{port}/v1/traces"
log.info("[OTLPSpanExporter] Connecting to OpenTelemetry Collector at %s", endpoint)
return OtelTrace(
- span_exporter=OTLPSpanExporter(endpoint=endpoint, headers={"Content-Type": "application/json"}),
+ span_exporter=OTLPSpanExporter(endpoint=endpoint),
tag_string=tag_string,
Review Comment:
`get_otel_tracer()`’s non-debug branch now instantiates `OTLPSpanExporter` without explicit headers, but the existing unit tests only exercise the debug=True path (ConsoleSpanExporter). Please add a test that sets `otel_debugging_on=False` and asserts `OTLPSpanExporter` is called with `endpoint=...` and no `headers` argument (so env-based `OTEL_EXPORTER_OTLP_HEADERS` can take effect).
##########
airflow/metrics/otel_logger.py:
##########
@@ -421,7 +421,6 @@ def get_otel_logger(cls) -> SafeOtelLogger:
PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=endpoint,
- headers={"Content-Type": "application/json"},
),
export_interval_millis=interval,
)
Review Comment:
`get_otel_logger()` now builds `OTLPMetricExporter(endpoint=endpoint)` without the previously hardcoded Content-Type header, but there doesn’t appear to be a unit test covering exporter construction/arguments (tests focus on SafeOtelLogger instrument behavior). Please add a test that patches `airflow.metrics.otel_logger.OTLPMetricExporter` and asserts it’s called with `endpoint=...` and without a `headers` override, to prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "MichaelRBlack (via GitHub)" <gi...@apache.org>.
MichaelRBlack commented on PR #63959:
URL: https://github.com/apache/airflow/pull/63959#issuecomment-4099915391
@potiuk Thanks for the review feedback. I've investigated all 3 CI failures and they are pre-existing on the `v2-11-stable` base branch, not caused by this PR's 2-line change (removing hardcoded `Content-Type` headers from OTLP exporters):
1. **Build docs (`--docs-only`)**: 297 toctree errors referencing nonexisting documents (e.g., `_api/airflow/decorators/index`, `_api/airflow/secrets/index`) in `public-airflow-interface.rst` — pre-existing in v2-11-stable.
2. **Spellcheck (`--spellcheck-only`)**: Undefined labels in `production-deployment.rst` (`howto/connection:gcp`, `howto/connection:gcp:impersonation`) — pre-existing in v2-11-stable.
3. **Provider tests** (`test_standard_task_runner.py::TestStandardTaskRunner::test_on_kill`): Flaky timeout failure (`AirflowTaskTimeout: Timeout, PID: 197`) — unrelated to OTLP exporters.
None of the files involved in these failures were touched by this PR. The branch is also already up to date with the latest `v2-11-stable` (no new commits on the base branch since this PR was opened).
Could you confirm whether these are known base-branch issues, or advise on the best way to proceed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #63959: Remove hardcoded Content-Type headers from OTLP exporters (backport #44346)
URL: https://github.com/apache/airflow/pull/63959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63959:
URL: https://github.com/apache/airflow/pull/63959#issuecomment-4416898392
@MichaelRBlack Converting to **draft** — this PR doesn't yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
- :x: **Unaddressed Copilot review** — 2 unresolved Copilot review threads have been waiting 39 days for a response: https://github.com/apache/airflow/pull/63959#discussion_r3025336685, https://github.com/apache/airflow/pull/63959#discussion_r3025336699. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is **not** a rejection — just an invitation to bring the PR up to standard. No rush.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
---
Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove hardcoded Content-Type headers from OTLP exporters (backport #44346) [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #63959:
URL: https://github.com/apache/airflow/pull/63959#issuecomment-4094598379
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org