You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by GitBox <gi...@apache.org> on 2026/04/20 08:57:58 UTC

[GH] (rocketmq/fix/otel-1.44-reusable-pool-race): Workflow run "Build and Run Tests by Bazel" failed!

The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git/fix/otel-1.44-reusable-pool-race has failed.
Run started by GitHub user Houlong66 (triggered by Houlong66).

Head commit for run:
cd983864741a0445d9e0ebbdeddadc02ad47d31a / 浩龙 <li...@alibaba-inc.com>
fix(metrics): mitigate OTel 1.44 OtlpGrpcMetricExporter pool race

OpenTelemetry Java 1.44.0 ~ 1.46.x ships OtlpGrpcMetricExporter with
MemoryMode.REUSABLE_DATA by default. The underlying
MetricReusableDataMarshaler.marshalerPool is a non-thread-safe
ArrayDeque accessed concurrently by the reader thread (poll) and the
OkHttp callback thread (add, via whenComplete). With
BatchSplittingMetricExporter issuing N concurrent sub-batch exports
per cycle, the pool races and leaks marshalers (~132 KiB each) until
OOM. Fixed upstream in 1.47.0 via open-telemetry/opentelemetry-java#7041
(ArrayDeque -> ConcurrentLinkedDeque).

- Bump OpenTelemetry to 1.47.0 in pom.xml so the upstream race fix is
  in effect.
- Default OtlpGrpcMetricExporter to MemoryMode.IMMUTABLE_DATA to
  preserve the pre-1.44 default behavior; exposed via
  brokerConfig.metricsExportOtelMemoryMode ("IMMUTABLE_DATA" /
  "REUSABLE_DATA", case-insensitive). Operators may opt in to
  REUSABLE_DATA when running on OTel >= 1.47.
- Cap concurrent in-flight sub-batches in BatchSplittingMetricExporter
  with a Semaphore controlled by
  brokerConfig.metricsExportBatchMaxConcurrent (default 4; set to 1
  to serialize and match pre-batch behavior; 0 or Integer.MAX_VALUE
  means unlimited).
- Add brokerConfig.metricsExportBatchSplitEnabled (default true) as
  an escape hatch to bypass BatchSplittingMetricExporter entirely,
  restoring the raw OtlpGrpcMetricExporter wiring.
- Defensively snapshot MetricData points before export to avoid
  ArrayIndexOutOfBoundsException in NumberDataPointMarshaler when
  async instrument callbacks mutate point collections during export.

Report URL: https://github.com/apache/rocketmq/actions/runs/24657530322

With regards,
GitHub Actions via GitBox