You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "Houlong66 (via GitHub)" <gi...@apache.org> on 2026/04/02 02:53:56 UTC
[I] [Bug] OTLP gRPC metrics export fails when payload exceeds 32MB limit due to high-cardinality metrics [rocketmq]
Houlong66 opened a new issue, #10240:
URL: https://github.com/apache/rocketmq/issues/10240
### Before Creating the Bug Report
- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
### Runtime platform environment
Linux / JDK 8+
### RocketMQ version
5.3.1
### Describe the Bug
When high-cardinality metrics (e.g., consumer lag with `consumer_group × topic` combinations) are exported via OTLP gRPC, the payload can exceed the gRPC 32MB message size limit, causing **all metrics to fail to export** with:
```
WARNING: Failed to export metrics. Server responded with gRPC status code 8.
Error message: grpc: received message larger than max (33568683 vs. 33554432)
```
Additionally, even when gzip compression is enabled, a single `MetricData` can contain tens of thousands of data points (e.g., 83,000+), producing a payload that exceeds backend per-RPC processing limits after decompression.
The OpenTelemetry Java SDK does not support automatic batch splitting for metrics export (see [opentelemetry-java#5394](https://github.com/open-telemetry/opentelemetry-java/issues/5394)).
### Steps to Reproduce
1. Configure a Broker with OTLP gRPC metrics export (`metricsExporterType=OTLP_GRPC`)
2. Have a large number of consumer groups and topics (e.g., 20,000+ topics, 20,000+ subscriptions)
3. Observe that metrics export fails with gRPC `RESOURCE_EXHAUSTED` or `UNKNOWN` errors
### What Did You Expect to See?
All metrics should be exported successfully regardless of cardinality.
### What Did You See Instead?
All metrics export fails every export cycle (default 60s), resulting in complete loss of observability.
### Additional Context
This issue is tracked upstream in the OpenTelemetry Java SDK as [opentelemetry-java#5394](https://github.com/open-telemetry/opentelemetry-java/issues/5394). The Python SDK already supports `max_export_batch_size`, but the Java SDK does not yet have this feature.
Fix: #10239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] OTLP gRPC metrics export fails when payload exceeds 32MB limit due to high-cardinality metrics [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins closed issue #10240: [Bug] OTLP gRPC metrics export fails when payload exceeds 32MB limit due to high-cardinality metrics
URL: https://github.com/apache/rocketmq/issues/10240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] OTLP gRPC metrics export fails when payload exceeds 32MB limit due to high-cardinality metrics [rocketmq]
Posted by "Houlong66 (via GitHub)" <gi...@apache.org>.
Houlong66 commented on issue #10240:
URL: https://github.com/apache/rocketmq/issues/10240#issuecomment-4177016432
## Follow-up: Fix for `ArrayIndexOutOfBoundsException` during OTLP export
In addition to the original gRPC 32MB payload limit issue, we discovered a related concurrent modification bug in production:
```
java.lang.ArrayIndexOutOfBoundsException: Index 9689 out of bounds for length 9689
at io.opentelemetry.exporter.internal.otlp.metrics.NumberDataPointMarshaler.createRepeated(NumberDataPointMarshaler.java:35)
```
**Root cause**: The OTel SDK's `NumberDataPointMarshaler.createRepeated` allocates an array based on `points.size()`, then iterates to fill it. If callback threads concurrently add data points between the `size()` call and iteration completion, the iterator yields more elements than the array can hold, causing AIOOBE.
**Fix**: Added a defensive snapshot step at the start of `BatchSplittingMetricExporter.export()` — all data point collections are copied into new `ArrayList` instances before being passed to the delegate exporter. This ensures the OTel SDK marshaling code always works with immutable snapshots, regardless of concurrent callback activity.
The snapshot is a shallow copy (references only, ~800KB for 100K data points), created once per export cycle (~60s) and immediately GC-eligible after export completes. Negligible overhead.
See commit: 81915011cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org