You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by yo...@apache.org on 2026/01/12 09:11:44 UTC
(bookkeeper) branch branch-4.17 updated (d95cf46e5a -> 6e6081c500)
This is an automated email from the ASF dual-hosted git repository.
yong pushed a change to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
from d95cf46e5a Exclude commons-collections from dependencies (#4689)
new 22fe67ffd2 [fix]compile error of the file native_io_jni.c for the env jdk11 & windows (#4665)
new 41bff23bd2 add rate limit for zk read rate in gc. (#4645)
new 67eacd0e57 Only stop Gc for the fill disk for DbLedgerStorage (#4661)
new 3c0cba1bd2 Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0 (#4690)
new 6e6081c500 Replace net.jpountz.lz4:lz4 with at.yawk.lz4:lz4-java (#4696)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../src/main/resources/LICENSE-all.bin.txt | 84 +++++++------
.../src/main/resources/LICENSE-bkctl.bin.txt | 41 ++++---
.../src/main/resources/LICENSE-server.bin.txt | 84 +++++++------
.../bookie/ScanAndCompareGarbageCollector.java | 7 ++
.../bookie/storage/ldb/DbLedgerStorage.java | 2 +-
.../ldb/SingleDirectoryDbLedgerStorage.java | 40 +++++-
.../bookkeeper/conf/ServerConfiguration.java | 19 +++
.../bookie/BookieStorageThresholdTest.java | 135 +++++++++++++++++++++
.../org/apache/bookkeeper/meta/GcLedgersTest.java | 48 ++++++++
conf/bk_server.conf | 3 +
.../src/main/native-io-jni/cpp/native_io_jni.c | 14 ++-
pom.xml | 33 ++---
shaded/distributedlog-core-shaded/pom.xml | 2 +-
site3/website/docs/reference/config.md | 1 +
.../bookkeeper/stats/otel/OtelMetricsProvider.java | 7 +-
stream/distributedlog/common/pom.xml | 4 +-
.../distributedlog/io/LZ4CompressionCodec.java | 8 +-
17 files changed, 395 insertions(+), 137 deletions(-)
(bookkeeper) 04/05: Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0 (#4690)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 3c0cba1bd23d3741c120ed7278e3ee036dc8c21f
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Dec 1 04:23:30 2025 +0200
Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0 (#4690)
* Upgrade OpenTelemetry to 1.56.0, intrumentation to 2.21.0 and semconv to 1.37.0
* Fix compatibility on Otel 1.56
* Add otel.contrib.version
* Align okhttp3 and okio versions
* Upgrade licenses
(cherry picked from commit d71cd1e84fa52497eaaf489856b8d019f4f42fa3)
---
.../src/main/resources/LICENSE-all.bin.txt | 76 +++++++++++-----------
.../src/main/resources/LICENSE-bkctl.bin.txt | 33 +++++-----
.../src/main/resources/LICENSE-server.bin.txt | 76 +++++++++++-----------
pom.xml | 25 ++++---
.../bookkeeper/stats/otel/OtelMetricsProvider.java | 7 +-
5 files changed, 110 insertions(+), 107 deletions(-)
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
index 853e338ace..a4c18fe3d8 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
@@ -325,41 +325,38 @@ Apache Software License, Version 2.
- lib/io.reactivex.rxjava3-rxjava-3.0.1.jar [51]
- lib/org.hdrhistogram-HdrHistogram-2.1.10.jar [52]
- lib/com.carrotsearch-hppc-0.9.1.jar [53]
-- lib/com.squareup.okhttp3-okhttp-4.12.0.jar [54]
-- lib/com.squareup.okio-okio-3.6.0.jar [54]
-- lib/com.squareup.okio-okio-jvm-3.6.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-api-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-api-incubator-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry-opentelemetry-context-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-exporter-common-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-exporter-otlp-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-exporter-otlp-common-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-exporter-prometheus-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry-opentelemetry-exporter-sender-okhttp-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-common-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-logs-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-trace-1.45.0.jar [55]
-- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry.semconv-opentelemetry-semconv-1.29.0-alpha.jar [55]
-- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-1.33.6.jar [59]
-- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-semconv-1.33.6-alpha.jar [59]
-- lib/io.opentelemetry.instrumentation-opentelemetry-runtime-telemetry-java8-1.33.6-alpha.jar [59]
-- lib/com.google.cloud.opentelemetry-detector-resources-support-0.33.0.jar [54]
-- lib/io.prometheus-prometheus-metrics-config-1.3.4.jar [60]
-- lib/io.prometheus-prometheus-metrics-exporter-common-1.3.4.jar [60]
-- lib/io.prometheus-prometheus-metrics-exporter-httpserver-1.3.4.jar [60]
-- lib/io.prometheus-prometheus-metrics-exposition-formats-1.3.4.jar [60]
-- lib/io.prometheus-prometheus-metrics-exposition-textformats-1.3.4.jar [60]
-- lib/io.prometheus-prometheus-metrics-model-1.3.4.jar [60]
+- lib/com.squareup.okhttp3-okhttp-jvm-5.3.1.jar [54]
+- lib/com.squareup.okio-okio-jvm-3.16.3.jar [54]
+- lib/io.opentelemetry-opentelemetry-api-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-api-incubator-1.56.0-alpha.jar [55]
+- lib/io.opentelemetry-opentelemetry-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-context-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-exporter-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-exporter-otlp-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-exporter-otlp-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-exporter-prometheus-1.56.0-alpha.jar [55]
+- lib/io.opentelemetry-opentelemetry-exporter-sender-okhttp-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-logs-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-trace-1.56.0.jar [55]
+- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.52.0-alpha.jar [55]
+- lib/io.opentelemetry.semconv-opentelemetry-semconv-1.37.0.jar [55]
+- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-2.21.0.jar [59]
+- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-incubator-2.21.0-alpha.jar [59]
+- lib/io.opentelemetry.instrumentation-opentelemetry-runtime-telemetry-java8-2.21.0-alpha.jar [59]
+- lib/com.google.cloud.opentelemetry-detector-resources-support-0.36.0.jar [55]
+- lib/io.prometheus-prometheus-metrics-config-1.3.10.jar [60]
+- lib/io.prometheus-prometheus-metrics-exporter-common-1.3.10.jar [60]
+- lib/io.prometheus-prometheus-metrics-exporter-httpserver-1.3.10.jar [60]
+- lib/io.prometheus-prometheus-metrics-exposition-formats-no-protobuf-1.3.10.jar [60]
+- lib/io.prometheus-prometheus-metrics-exposition-textformats-1.3.10.jar [60]
+- lib/io.prometheus-prometheus-metrics-model-1.3.10.jar [60]
- lib/org.jetbrains-annotations-13.0.jar [56]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-1.8.21.jar [56]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.8.21.jar [56]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.21.jar [56]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.21.jar [56]
+- lib/org.jetbrains.kotlin-kotlin-stdlib-2.2.21.jar [56]
[1] Source available at https://github.com/FasterXML/jackson-annotations/tree/jackson-annotations-2.17.1
[2] Source available at https://github.com/FasterXML/jackson-core/tree/jackson-core-2.17.1
@@ -409,12 +406,13 @@ Apache Software License, Version 2.
[51] Source available at https://github.com/ReactiveX/RxJava/tree/v3.0.1
[52] Source available at https://github.com/HdrHistogram/HdrHistogram/tree/HdrHistogram-2.1.10
[53] Source available at https://github.com/carrotsearch/hppc/tree/0.9.1
-[54] Source available at https://github.com/square/okio/releases/tag/parent-3.6.0
-[55] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.45.0
-[56] Source available at https://github.com/JetBrains/kotlin/releases/tag/v1.8.21
+[54] Source available at https://github.com/square/okio/releases/tag/parent-3.16.3
+[55] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.56.0
+[56] Source available at https://github.com/JetBrains/kotlin/releases/tag/v2.2.21
+[57] Source available at https://github.com/LMAX-Exchange/disruptor/releases/tag/4.0.0
[58] Source available at https://github.com/apache/commons-lang/tree/rel/commons-lang-3.17.0
-[59] Source available at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/v1.33.6
-[60] Source available at https://github.com/prometheus/client_java/tree/v1.3.4
+[59] Source available at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/v2.21.0
+[60] Source available at https://github.com/prometheus/client_java/tree/v1.3.10
[61] Source available at https://github.com/apache/commons-text/tree/rel/commons-text-1.13.1
[62] Source available at https://github.com/apache/commons-beanutils/tree/rel/commons-beanutils-1.11.0
------------------------------------------------------------------------------------
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
index 28a27b6661..962268f745 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
@@ -292,21 +292,21 @@ Apache Software License, Version 2.
- lib/org.xerial.snappy-snappy-java-1.1.10.5.jar [50]
- lib/io.reactivex.rxjava3-rxjava-3.0.1.jar [51]
- lib/com.carrotsearch-hppc-0.9.1.jar [52]
-- lib/io.opentelemetry-opentelemetry-api-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-context-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-api-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-api-incubator-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry-opentelemetry-context-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-common-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-logs-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-trace-1.45.0.jar [55]
-- lib/io.opentelemetry-opentelemetry-api-incubator-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.45.0-alpha.jar [55]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.45.0.jar [55]
-- lib/com.google.cloud.opentelemetry-detector-resources-support-0.33.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-api-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-common-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-context-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-api-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-context-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-common-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-logs-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.56.0.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-trace-1.56.0.jar [55]
+- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.52.0-alpha.jar [55]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.56.0.jar [55]
+- lib/com.google.cloud.opentelemetry-detector-resources-support-0.36.0.jar [54]
[1] Source available at https://github.com/FasterXML/jackson-annotations/tree/jackson-annotations-2.17.1
[2] Source available at https://github.com/FasterXML/jackson-core/tree/jackson-core-2.17.1
@@ -347,7 +347,8 @@ Apache Software License, Version 2.
[50] Source available at https://github.com/xerial/snappy-java/releases/tag/v1.1.10.5
[51] Source available at https://github.com/ReactiveX/RxJava/tree/v3.0.1
[52] Source available at https://github.com/carrotsearch/hppc/tree/0.9.1
-[54] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.45.0
+[53] Source available at https://github.com/LMAX-Exchange/disruptor/releases/tag/4.0.0
+[54] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.56.0
[55] Source available at https://github.com/apache/commons-lang/tree/rel/commons-lang-3.17.0
[56] Source available at https://github.com/apache/commons-text/tree/rel/commons-text-1.13.1
[57] Source available at https://github.com/apache/commons-beanutils/tree/rel/commons-beanutils-1.11.0
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
index c05e3c57d6..32f518ac32 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
@@ -321,41 +321,38 @@ Apache Software License, Version 2.
- lib/org.xerial.snappy-snappy-java-1.1.10.5.jar [50]
- lib/io.reactivex.rxjava3-rxjava-3.0.1.jar [51]
- lib/com.carrotsearch-hppc-0.9.1.jar [52]
-- lib/com.squareup.okhttp3-okhttp-4.12.0.jar [53]
-- lib/com.squareup.okio-okio-3.6.0.jar [53]
-- lib/com.squareup.okio-okio-jvm-3.6.0.jar [53]
-- lib/io.opentelemetry-opentelemetry-api-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-api-incubator-1.45.0-alpha.jar [54]
-- lib/io.opentelemetry-opentelemetry-context-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-exporter-common-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-exporter-otlp-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-exporter-otlp-common-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-exporter-prometheus-1.45.0-alpha.jar [54]
-- lib/io.opentelemetry-opentelemetry-exporter-sender-okhttp-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-common-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-logs-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.45.0.jar [54]
-- lib/io.opentelemetry-opentelemetry-sdk-trace-1.45.0.jar [54]
-- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-1.33.6.jar [58]
-- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-semconv-1.33.6-alpha.jar [58]
-- lib/io.opentelemetry.instrumentation-opentelemetry-runtime-telemetry-java8-1.33.6-alpha.jar [58]
-- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.45.0-alpha.jar [54]
-- lib/io.opentelemetry.semconv-opentelemetry-semconv-1.29.0-alpha.jar [54]
-- lib/com.google.cloud.opentelemetry-detector-resources-support-0.33.0.jar [54]
-- lib/io.prometheus-prometheus-metrics-config-1.3.4.jar [59]
-- lib/io.prometheus-prometheus-metrics-exporter-common-1.3.4.jar [59]
-- lib/io.prometheus-prometheus-metrics-exporter-httpserver-1.3.4.jar [59]
-- lib/io.prometheus-prometheus-metrics-exposition-formats-1.3.4.jar [59]
-- lib/io.prometheus-prometheus-metrics-exposition-textformats-1.3.4.jar [59]
-- lib/io.prometheus-prometheus-metrics-model-1.3.4.jar [59]
+- lib/com.squareup.okhttp3-okhttp-jvm-5.3.1.jar [53]
+- lib/com.squareup.okio-okio-jvm-3.16.3.jar [53]
+- lib/io.opentelemetry-opentelemetry-api-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-api-incubator-1.56.0-alpha.jar [54]
+- lib/io.opentelemetry-opentelemetry-common-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-context-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-exporter-common-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-exporter-otlp-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-exporter-otlp-common-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-exporter-prometheus-1.56.0-alpha.jar [54]
+- lib/io.opentelemetry-opentelemetry-exporter-sender-okhttp-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-common-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-extension-autoconfigure-spi-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-logs-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-metrics-1.56.0.jar [54]
+- lib/io.opentelemetry-opentelemetry-sdk-trace-1.56.0.jar [54]
+- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-2.21.0.jar [58]
+- lib/io.opentelemetry.instrumentation-opentelemetry-instrumentation-api-incubator-2.21.0-alpha.jar [58]
+- lib/io.opentelemetry.instrumentation-opentelemetry-runtime-telemetry-java8-2.21.0-alpha.jar [58]
+- lib/io.opentelemetry.contrib-opentelemetry-gcp-resources-1.52.0-alpha.jar [54]
+- lib/io.opentelemetry.semconv-opentelemetry-semconv-1.37.0.jar [54]
+- lib/com.google.cloud.opentelemetry-detector-resources-support-0.36.0.jar [54]
+- lib/io.prometheus-prometheus-metrics-config-1.3.10.jar [59]
+- lib/io.prometheus-prometheus-metrics-exporter-common-1.3.10.jar [59]
+- lib/io.prometheus-prometheus-metrics-exporter-httpserver-1.3.10.jar [59]
+- lib/io.prometheus-prometheus-metrics-exposition-formats-no-protobuf-1.3.10.jar [59]
+- lib/io.prometheus-prometheus-metrics-exposition-textformats-1.3.10.jar [59]
+- lib/io.prometheus-prometheus-metrics-model-1.3.10.jar [59]
- lib/org.jetbrains-annotations-13.0.jar [55]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-1.8.21.jar [55]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.8.21.jar [55]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.8.21.jar [55]
-- lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.8.21.jar [55]
+- lib/org.jetbrains.kotlin-kotlin-stdlib-2.2.21.jar [55]
[1] Source available at https://github.com/FasterXML/jackson-annotations/tree/jackson-annotations-2.17.1
[2] Source available at https://github.com/FasterXML/jackson-core/tree/jackson-core-2.17.1
@@ -404,12 +401,13 @@ Apache Software License, Version 2.
[50] Source available at https://github.com/xerial/snappy-java/releases/tag/v1.1.10.5
[51] Source available at https://github.com/ReactiveX/RxJava/tree/v3.0.1
[52] Source available at https://github.com/carrotsearch/hppc/tree/0.9.1
-[53] Source available at https://github.com/square/okio/releases/tag/parent-3.6.0
-[54] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.45.0
-[55] Source available at https://github.com/JetBrains/kotlin/releases/tag/v1.8.21
+[53] Source available at https://github.com/square/okio/releases/tag/parent-3.16.3
+[54] Source available at https://github.com/open-telemetry/opentelemetry-java/releases/tag/v1.56.0
+[55] Source available at https://github.com/JetBrains/kotlin/releases/tag/v2.2.21
+[56] Source available at https://github.com/LMAX-Exchange/disruptor/releases/tag/4.0.0
[57] Source available at https://github.com/apache/commons-lang/tree/rel/commons-lang-3.17.0
-[58] Source available at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/v1.33.6
-[59] Source available at https://github.com/prometheus/client_java/tree/v1.3.4
+[58] Source available at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/v2.21.0
+[59] Source available at https://github.com/prometheus/client_java/tree/v1.3.10
[60] Source available at https://github.com/apache/commons-text/tree/rel/commons-text-1.13.1
[61] Source available at https://github.com/apache/commons-beanutils/tree/rel/commons-beanutils-1.11.0
------------------------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index e463cab3cc..b079e79b1a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -202,9 +202,13 @@
<forkCount.variable>1</forkCount.variable>
<servlet-api.version>4.0.0</servlet-api.version>
<rxjava.version>3.0.1</rxjava.version>
- <otel.version>1.45.0</otel.version>
- <otel.instrumentation.version>1.33.6</otel.instrumentation.version>
- <otel.semconv.version>1.29.0-alpha</otel.semconv.version>
+ <otel.version>1.56.0</otel.version>
+ <otel.instrumentation.version>2.21.0</otel.instrumentation.version>
+ <otel.semconv.version>1.37.0</otel.semconv.version>
+ <otel.contrib.version>1.52.0</otel.contrib.version>
+ <okhttp3.version>5.3.1</okhttp3.version>
+ <!-- use okio version that matches the okhttp3 version -->
+ <okio.version>3.16.3</okio.version>
<UBUNTU_MIRROR>http://archive.ubuntu.com/ubuntu/</UBUNTU_MIRROR>
<UBUNTU_SECURITY_MIRROR>http://security.ubuntu.com/ubuntu/</UBUNTU_SECURITY_MIRROR>
</properties>
@@ -468,21 +472,22 @@
<version>${failsafe.version}</version>
</dependency>
- <!-- override otel's okhttp 4.11.0 for now, wait for otel update -->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp-bom</artifactId>
- <version>4.12.0</version>
+ <version>${okhttp3.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
- <!-- okhttp 4.12.0 use kotlin stdlib 1.8.21 -->
<dependency>
- <groupId>org.jetbrains.kotlin</groupId>
- <artifactId>kotlin-stdlib-common</artifactId>
- <version>1.8.21</version>
+ <groupId>com.squareup.okio</groupId>
+ <artifactId>okio-bom</artifactId>
+ <version>${okio.version}</version>
+ <type>pom</type>
+ <scope>import</scope>
</dependency>
+
<!-- rocksdb dependencies -->
<dependency>
<groupId>org.rocksdb</groupId>
@@ -655,7 +660,7 @@
<dependency>
<groupId>io.opentelemetry.contrib</groupId>
<artifactId>opentelemetry-gcp-resources</artifactId>
- <version>${otel.version}-alpha</version>
+ <version>${otel.contrib.version}-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
diff --git a/stats/bookkeeper-stats-providers/otel-metrics-provider/src/main/java/org/apache/bookkeeper/stats/otel/OtelMetricsProvider.java b/stats/bookkeeper-stats-providers/otel-metrics-provider/src/main/java/org/apache/bookkeeper/stats/otel/OtelMetricsProvider.java
index 70a845965f..817db0aa78 100644
--- a/stats/bookkeeper-stats-providers/otel-metrics-provider/src/main/java/org/apache/bookkeeper/stats/otel/OtelMetricsProvider.java
+++ b/stats/bookkeeper-stats-providers/otel-metrics-provider/src/main/java/org/apache/bookkeeper/stats/otel/OtelMetricsProvider.java
@@ -17,16 +17,17 @@
package org.apache.bookkeeper.stats.otel;
// CHECKSTYLE.OFF: IllegalImport
+
import io.netty.util.internal.PlatformDependent;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.Meter;
-import io.opentelemetry.instrumentation.runtimemetrics.java8.BufferPools;
import io.opentelemetry.instrumentation.runtimemetrics.java8.Classes;
import io.opentelemetry.instrumentation.runtimemetrics.java8.Cpu;
import io.opentelemetry.instrumentation.runtimemetrics.java8.GarbageCollector;
import io.opentelemetry.instrumentation.runtimemetrics.java8.MemoryPools;
import io.opentelemetry.instrumentation.runtimemetrics.java8.Threads;
+import io.opentelemetry.instrumentation.runtimemetrics.java8.internal.ExperimentalBufferPools;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
import io.opentelemetry.sdk.metrics.Aggregation;
import io.opentelemetry.sdk.metrics.InstrumentSelector;
@@ -96,11 +97,11 @@ public class OtelMetricsProvider implements StatsProvider {
if (exposeDefaultJVMMetrics) {
// Include standard JVM stats
MemoryPools.registerObservers(openTelemetry);
- BufferPools.registerObservers(openTelemetry);
+ ExperimentalBufferPools.registerObservers(openTelemetry);
Classes.registerObservers(openTelemetry);
Cpu.registerObservers(openTelemetry);
Threads.registerObservers(openTelemetry);
- GarbageCollector.registerObservers(openTelemetry);
+ GarbageCollector.registerObservers(openTelemetry, true);
meter.gaugeBuilder("process.runtime.jvm.memory.direct_bytes_used")
.buildWithCallback(odm -> odm.record(getDirectMemoryUsage.get()));
(bookkeeper) 05/05: Replace net.jpountz.lz4:lz4 with at.yawk.lz4:lz4-java (#4696)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 6e6081c500dca2a4bdcd0ebf5f5a203d67df10cd
Author: Masahiro Sakamoto <ma...@lycorp.co.jp>
AuthorDate: Mon Jan 12 17:07:10 2026 +0900
Replace net.jpountz.lz4:lz4 with at.yawk.lz4:lz4-java (#4696)
(cherry picked from commit b88fb5f0390a37947b33f1ba151de820f10452c5)
---
bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt | 8 ++++----
bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt | 8 ++++----
bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt | 8 ++++----
pom.xml | 8 ++++----
shaded/distributedlog-core-shaded/pom.xml | 2 +-
stream/distributedlog/common/pom.xml | 4 ++--
.../java/org/apache/distributedlog/io/LZ4CompressionCodec.java | 8 ++++----
7 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
index a4c18fe3d8..61b2918b56 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-all.bin.txt
@@ -276,7 +276,7 @@ Apache Software License, Version 2.
- lib/com.beust-jcommander-1.82.jar [24]
- lib/com.yahoo.datasketches-memory-0.8.3.jar [25]
- lib/com.yahoo.datasketches-sketches-core-0.8.3.jar [25]
-- lib/net.jpountz.lz4-lz4-1.3.0.jar [26]
+- lib/at.yawk.lz4-lz4-java-1.10.2.jar [26]
- lib/com.google.api.grpc-proto-google-common-protos-2.51.0.jar [28]
- lib/com.google.code.gson-gson-2.11.0.jar [29]
- lib/io.opencensus-opencensus-api-0.31.1.jar [30]
@@ -382,7 +382,7 @@ Apache Software License, Version 2.
[23] Source available at https://github.com/facebook/rocksdb/tree/v7.10.2
[24] Source available at https://github.com/cbeust/jcommander/tree/1.82
[25] Source available at https://github.com/DataSketches/sketches-core/tree/sketches-0.8.3
-[26] Source available at https://github.com/lz4/lz4-java/tree/1.3.0
+[26] Source available at https://github.com/yawkat/lz4-java/tree/v1.10.2
[28] Source available at https://github.com/googleapis/java-common-protos/tree/v2.51.0
[29] Source available at https://github.com/google/gson/tree/gson-parent-2.11.0
[30] Source available at https://github.com/census-instrumentation/opencensus-java/tree/v0.31.1
@@ -502,13 +502,13 @@ decoding data in LZF format, written by Tatu Saloranta. It can be obtained at:
* HOMEPAGE:
* https://github.com/ning/compress
-lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4', a LZ4 Java compression
+lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4-java', a LZ4 Java compression
and decompression library written by Adrien Grand. It can be obtained at:
* LICENSE:
* deps/netty/LICENSE.lz4.txt (Apache License 2.0)
* HOMEPAGE:
- * https://github.com/jpountz/lz4-java
+ * https://github.com/yawkat/lz4-java
lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lzma-java', a LZMA Java compression
and decompression library, which can be obtained at:
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
index 962268f745..5c295e860d 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
@@ -247,7 +247,7 @@ Apache Software License, Version 2.
- lib/org.apache.zookeeper-zookeeper-jute-3.9.3.jar [20]
- lib/org.apache.zookeeper-zookeeper-3.9.3-tests.jar [20]
- lib/com.beust-jcommander-1.82.jar [23]
-- lib/net.jpountz.lz4-lz4-1.3.0.jar [25]
+- lib/at.yawk.lz4-lz4-java-1.10.2.jar [25]
- lib/com.google.api.grpc-proto-google-common-protos-2.51.0.jar [27]
- lib/com.google.code.gson-gson-2.11.0.jar [28]
- lib/io.opencensus-opencensus-api-0.31.1.jar [29]
@@ -324,7 +324,7 @@ Apache Software License, Version 2.
[19] Source available at https://github.com/apache/commons-lang/tree/LANG_3_6
[20] Source available at https://github.com/apache/zookeeper/tree/release-3.8.0
[23] Source available at https://github.com/cbeust/jcommander/tree/1.82
-[25] Source available at https://github.com/lz4/lz4-java/tree/1.3.0
+[25] Source available at https://github.com/yawkat/lz4-java/tree/v1.10.2
[27] Source available at https://github.com/googleapis/java-common-protos/tree/v2.51.0
[28] Source available at https://github.com/google/gson/tree/gson-parent-2.11.0
[29] Source available at https://github.com/census-instrumentation/opencensus-java/tree/v0.31.1
@@ -439,13 +439,13 @@ decoding data in LZF format, written by Tatu Saloranta. It can be obtained at:
* HOMEPAGE:
* https://github.com/ning/compress
-lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4', a LZ4 Java compression
+lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4-java', a LZ4 Java compression
and decompression library written by Adrien Grand. It can be obtained at:
* LICENSE:
* deps/netty/LICENSE.lz4.txt (Apache License 2.0)
* HOMEPAGE:
- * https://github.com/jpountz/lz4-java
+ * https://github.com/yawkat/lz4-java
lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lzma-java', a LZMA Java compression
and decompression library, which can be obtained at:
diff --git a/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt b/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
index 32f518ac32..116bc5bc7d 100644
--- a/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
+++ b/bookkeeper-dist/src/main/resources/LICENSE-server.bin.txt
@@ -276,7 +276,7 @@ Apache Software License, Version 2.
- lib/com.beust-jcommander-1.82.jar [24]
- lib/com.yahoo.datasketches-memory-0.8.3.jar [25]
- lib/com.yahoo.datasketches-sketches-core-0.8.3.jar [25]
-- lib/net.jpountz.lz4-lz4-1.3.0.jar [26]
+- lib/at.yawk.lz4-lz4-java-1.10.2.jar [26]
- lib/com.google.api.grpc-proto-google-common-protos-2.51.0.jar [28]
- lib/com.google.code.gson-gson-2.11.0.jar [29]
- lib/io.opencensus-opencensus-api-0.31.1.jar [30]
@@ -378,7 +378,7 @@ Apache Software License, Version 2.
[23] Source available at https://github.com/facebook/rocksdb/tree/v7.10.2
[24] Source available at https://github.com/cbeust/jcommander/tree/1.82
[25] Source available at https://github.com/DataSketches/sketches-core/tree/sketches-0.8.3
-[26] Source available at https://github.com/lz4/lz4-java/tree/1.3.0
+[26] Source available at https://github.com/yawkat/lz4-java/tree/v1.10.2
[28] Source available at https://github.com/googleapis/java-common-protos/tree/v2.51.0
[29] Source available at https://github.com/google/gson/tree/gson-parent-2.11.0
[30] Source available at https://github.com/census-instrumentation/opencensus-java/tree/v0.31.1
@@ -497,13 +497,13 @@ decoding data in LZF format, written by Tatu Saloranta. It can be obtained at:
* HOMEPAGE:
* https://github.com/ning/compress
-lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4', a LZ4 Java compression
+lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lz4-java', a LZ4 Java compression
and decompression library written by Adrien Grand. It can be obtained at:
* LICENSE:
* deps/netty/LICENSE.lz4.txt (Apache License 2.0)
* HOMEPAGE:
- * https://github.com/jpountz/lz4-java
+ * https://github.com/yawkat/lz4-java
lib/io.netty-netty-codec-4.1.121.Final.jar optionally depends on 'lzma-java', a LZMA Java compression
and decompression library, which can be obtained at:
diff --git a/pom.xml b/pom.xml
index b079e79b1a..a8e1701419 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,7 +154,7 @@
<libthrift.version>0.14.2</libthrift.version>
<lombok.version>1.18.30</lombok.version>
<log4j.version>2.18.0</log4j.version>
- <lz4.version>1.3.0</lz4.version>
+ <lz4-java.version>1.10.2</lz4-java.version>
<mockito.version>4.11.0</mockito.version>
<netty.version>4.1.121.Final</netty.version>
<netty-iouring.version>0.0.26.Final</netty-iouring.version>
@@ -337,9 +337,9 @@
<!-- compression libs -->
<dependency>
- <groupId>net.jpountz.lz4</groupId>
- <artifactId>lz4</artifactId>
- <version>${lz4.version}</version>
+ <groupId>at.yawk.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>${lz4-java.version}</version>
</dependency>
<!-- yaml dependencies -->
diff --git a/shaded/distributedlog-core-shaded/pom.xml b/shaded/distributedlog-core-shaded/pom.xml
index 899279fd1b..1501a877a3 100644
--- a/shaded/distributedlog-core-shaded/pom.xml
+++ b/shaded/distributedlog-core-shaded/pom.xml
@@ -74,7 +74,7 @@
<include>com.google.guava:failureaccess</include>
<include>com.google.guava:guava</include>
<include>com.google.protobuf:protobuf-java</include>
- <include>net.jpountz.lz4:lz4</include>
+ <include>at.yawk.lz4:lz4-java</include>
<include>org.apache.bookkeeper:bookkeeper-common</include>
<include>org.apache.bookkeeper:bookkeeper-common-allocator</include>
<include>org.apache.bookkeeper:cpu-affinity</include>
diff --git a/stream/distributedlog/common/pom.xml b/stream/distributedlog/common/pom.xml
index b5b312ea57..fcfdff6328 100644
--- a/stream/distributedlog/common/pom.xml
+++ b/stream/distributedlog/common/pom.xml
@@ -70,8 +70,8 @@
<artifactId>netty-buffer</artifactId>
</dependency>
<dependency>
- <groupId>net.jpountz.lz4</groupId>
- <artifactId>lz4</artifactId>
+ <groupId>at.yawk.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
diff --git a/stream/distributedlog/common/src/main/java/org/apache/distributedlog/io/LZ4CompressionCodec.java b/stream/distributedlog/common/src/main/java/org/apache/distributedlog/io/LZ4CompressionCodec.java
index 48439d23a1..a02322f4cc 100644
--- a/stream/distributedlog/common/src/main/java/org/apache/distributedlog/io/LZ4CompressionCodec.java
+++ b/stream/distributedlog/common/src/main/java/org/apache/distributedlog/io/LZ4CompressionCodec.java
@@ -25,7 +25,7 @@ import io.netty.buffer.PooledByteBufAllocator;
import java.nio.ByteBuffer;
import net.jpountz.lz4.LZ4Compressor;
import net.jpountz.lz4.LZ4Factory;
-import net.jpountz.lz4.LZ4FastDecompressor;
+import net.jpountz.lz4.LZ4SafeDecompressor;
/**
* An {@code lz4} based {@link CompressionCodec} implementation.
@@ -44,7 +44,7 @@ public class LZ4CompressionCodec implements CompressionCodec {
// Used for compression
private static final LZ4Compressor compressor = factory.fastCompressor();
// Used to decompress when the size of the output is known
- private static final LZ4FastDecompressor decompressor = factory.fastDecompressor();
+ private static final LZ4SafeDecompressor decompressor = factory.safeDecompressor();
@Override
public ByteBuf compress(ByteBuf uncompressed, int headerLen) {
@@ -69,7 +69,7 @@ public class LZ4CompressionCodec implements CompressionCodec {
}
@Override
- // length parameter is ignored here because of the way the fastDecompressor works.
+ // length parameter is ignored here because of the way the safeDecompressor works.
public ByteBuf decompress(ByteBuf compressed, int decompressedSize) {
checkNotNull(compressed);
checkArgument(compressed.readableBytes() >= 0);
@@ -80,7 +80,7 @@ public class LZ4CompressionCodec implements CompressionCodec {
ByteBuffer compressedNio = compressed.nioBuffer(compressed.readerIndex(), compressed.readableBytes());
decompressor.decompress(
- compressedNio, compressedNio.position(),
+ compressedNio, compressedNio.position(), compressedNio.remaining(),
uncompressedNio, uncompressedNio.position(), uncompressedNio.remaining());
uncompressed.writerIndex(decompressedSize);
return uncompressed;
(bookkeeper) 02/05: add rate limit for zk read rate in gc. (#4645)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 41bff23bd293a5daffc0072aca5cceaa967d2a1b
Author: Wenzhi Feng <th...@apache.org>
AuthorDate: Tue Aug 12 16:37:25 2025 +0800
add rate limit for zk read rate in gc. (#4645)
* add rate limit for gc.
* fix checkstyle.
* fix conf name and acqurie.
* rename gcZkOpRateLimit to gcMetadataOpRateLimit.
* rename gcZkOpRateLimit to gcMetadataOpRateLimit.
* add return label.
* add test code.
* document conf.
---------
Co-authored-by: fengwenzhi <fe...@bigo.sg>
(cherry picked from commit 417ff1651fed3d37abc4b3d4e57f8eac950dce9e)
---
.../bookie/ScanAndCompareGarbageCollector.java | 7 ++++
.../bookkeeper/conf/ServerConfiguration.java | 19 +++++++++
.../org/apache/bookkeeper/meta/GcLedgersTest.java | 48 ++++++++++++++++++++++
conf/bk_server.conf | 3 ++
site3/website/docs/reference/config.md | 1 +
5 files changed, 78 insertions(+)
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
index 34a54c3e4e..4cdedb36b2 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
@@ -24,6 +24,7 @@ package org.apache.bookkeeper.bookie;
import static org.apache.bookkeeper.common.concurrent.FutureUtils.result;
import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.RateLimiter;
import java.io.IOException;
import java.net.URI;
import java.util.List;
@@ -84,6 +85,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
private int activeLedgerCounter;
private StatsLogger statsLogger;
private final int maxConcurrentRequests;
+ private final RateLimiter gcMetadataOpRateLimiter;
public ScanAndCompareGarbageCollector(LedgerManager ledgerManager, CompactableLedgerStorage ledgerStorage,
ServerConfiguration conf, StatsLogger statsLogger) throws IOException {
@@ -103,6 +105,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
enableGcOverReplicatedLedger, gcOverReplicatedLedgerIntervalMillis, maxConcurrentRequests);
verifyMetadataOnGc = conf.getVerifyMetadataOnGC();
+ this.gcMetadataOpRateLimiter = RateLimiter.create(conf.getGcMetadataOpRateLimit());
this.activeLedgerCounter = 0;
}
@@ -153,6 +156,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
Versioned<LedgerMetadata> metadata = null;
while (!done) {
start = end + 1;
+ gcMetadataOpRateLimiter.acquire();
if (ledgerRangeIterator.hasNext()) {
LedgerRange lRange = ledgerRangeIterator.next();
ledgersInMetadata = lRange.getLedgers();
@@ -175,6 +179,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
metadata = null;
int rc = BKException.Code.OK;
try {
+ gcMetadataOpRateLimiter.acquire();
metadata = result(ledgerManager.readLedgerMetadata(bkLid), zkOpTimeoutMs,
TimeUnit.MILLISECONDS);
} catch (BKException | TimeoutException e) {
@@ -236,6 +241,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
// check ledger ensembles before creating lock nodes.
// this is to reduce the number of lock node creations and deletions in ZK.
// the ensemble check is done again after the lock node is created.
+ gcMetadataOpRateLimiter.acquire();
Versioned<LedgerMetadata> preCheckMetadata = ledgerManager.readLedgerMetadata(ledgerId).get();
if (!isNotBookieIncludedInLedgerEnsembles(preCheckMetadata)) {
latch.countDown();
@@ -261,6 +267,7 @@ public class ScanAndCompareGarbageCollector implements GarbageCollector {
// current bookie again and, in that case, we cannot remove the ledger from local storage
lum.acquireUnderreplicatedLedger(ledgerId);
semaphore.acquire();
+ gcMetadataOpRateLimiter.acquire();
ledgerManager.readLedgerMetadata(ledgerId)
.whenComplete((metadata, exception) -> {
try {
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
index b62922db40..4d9eabb241 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
@@ -114,6 +114,7 @@ public class ServerConfiguration extends AbstractConfiguration<ServerConfigurati
protected static final String GC_OVERREPLICATED_LEDGER_WAIT_TIME = "gcOverreplicatedLedgerWaitTime";
protected static final String GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_REQUESTS =
"gcOverreplicatedLedgerMaxConcurrentRequests";
+ protected static final String GC_METADATA_OP_RATE_LIMIT = "gcMetadataOpRateLimit";
protected static final String USE_TRANSACTIONAL_COMPACTION = "useTransactionalCompaction";
protected static final String VERIFY_METADATA_ON_GC = "verifyMetadataOnGC";
protected static final String GC_ENTRYLOGMETADATA_CACHE_ENABLED = "gcEntryLogMetadataCacheEnabled";
@@ -481,6 +482,24 @@ public class ServerConfiguration extends AbstractConfiguration<ServerConfigurati
return this;
}
+ /**
+ * Get the rate limit of metadata operations in garbage collection.
+ * @return rate limit of metadata operations in garbage collection
+ */
+ public int getGcMetadataOpRateLimit() {
+ return this.getInt(GC_METADATA_OP_RATE_LIMIT, 1000);
+ }
+
+ /**
+ * Set the rate limit of metadata operations in garbage collection.
+ * @param gcRateLimit
+ * @return server configuration
+ */
+ public ServerConfiguration setGcMetadataOpRateLimit(int gcRateLimit) {
+ this.setProperty(GC_METADATA_OP_RATE_LIMIT, Integer.toString(gcRateLimit));
+ return this;
+ }
+
/**
* Get whether to use transactional compaction and using a separate log for compaction or not.
*
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java
index fec74a8202..058dc8ce34 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java
@@ -36,6 +36,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.EnumSet;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
@@ -342,6 +343,53 @@ public class GcLedgersTest extends LedgerManagerTestCase {
assertEquals("Should have cleaned first ledger" + first, (long) first, (long) cleaned.get(0));
}
+
+ /**
+ * Verifies that the garbage collector respects the configured rate limit for metadata operations.
+ * @throws Exception
+ */
+ @Test
+ public void testGcMetadataOpRateLimit() throws Exception {
+ int numLedgers = 2000;
+ int numRemovedLedgers = 800;
+ final Set<Long> createdLedgers = new HashSet<Long>();
+ createLedgers(numLedgers, createdLedgers);
+
+ ServerConfiguration conf = new ServerConfiguration(baseConf);
+ int customRateLimit = 200;
+ conf.setGcMetadataOpRateLimit(customRateLimit);
+ // set true to verify metadata on gc
+ conf.setVerifyMetadataOnGc(true);
+
+ final GarbageCollector garbageCollector = new ScanAndCompareGarbageCollector(
+ getLedgerManager(), new MockLedgerStorage(), conf, NullStatsLogger.INSTANCE);
+
+ // delete created ledgers to simulate the garbage collection scenario
+ Iterator<Long> createdLedgersIterator = createdLedgers.iterator();
+ for (int i = 0; i < numRemovedLedgers && createdLedgersIterator.hasNext(); i++) {
+ long ledgerId = createdLedgersIterator.next();
+ try {
+ removeLedger(ledgerId);
+ } catch (Exception e) {
+ LOG.error("Failed to remove ledger {}", ledgerId, e);
+ }
+ }
+
+ long startTime = System.currentTimeMillis();
+ garbageCollector.gc(new GarbageCollector.GarbageCleaner() {
+ @Override
+ public void clean(long ledgerId) {
+ }
+ });
+ long endTime = System.currentTimeMillis();
+ long duration = endTime - startTime;
+ long minExpectedTime = (numRemovedLedgers * 1000L) / customRateLimit;
+
+ LOG.info("GC operation with rate limit {} took {} ms, theoretical minimum time: {} ms",
+ customRateLimit, duration, minExpectedTime);
+ assertTrue("GC operation should be rate limited", duration >= minExpectedTime * 0.7);
+ }
+
/*
* in this scenario no ledger is created, so ledgeriterator's hasNext call would return false and next would be
* null. GarbageCollector.gc is expected to behave normally
diff --git a/conf/bk_server.conf b/conf/bk_server.conf
index 3ea2a377c2..ecfa6b491b 100755
--- a/conf/bk_server.conf
+++ b/conf/bk_server.conf
@@ -601,6 +601,9 @@ ledgerDirectories=/tmp/bk-data
# interval if there is enough disk capacity.
# gcWaitTime=1000
+# The rate limit of metadata operations in garbage collection.
+#gcMetadataOpRateLimit=1000
+
# How long the interval to trigger next garbage collection of overreplicated
# ledgers, in milliseconds [Default: 1 day]. This should not be run very frequently
# since we read the metadata for all the ledgers on the bookie from zk
diff --git a/site3/website/docs/reference/config.md b/site3/website/docs/reference/config.md
index a7ea81f012..339b3c5fa4 100644
--- a/site3/website/docs/reference/config.md
+++ b/site3/website/docs/reference/config.md
@@ -189,6 +189,7 @@ The table below lists parameters that you can set to configure bookies. All conf
| Parameter | Description | Default
| --------- | ----------- | ------- |
| gcWaitTime | How long the interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in background, too frequent gc will heart performance. It is better to give a higher number of gc interval if there is enough disk capacity. | 1000 |
+| gcMetadataOpRateLimit | Rate limit for metadata operations in garbage collection, in operations per second. This is used to limit the rate of metadata operations during garbage collection to avoid overwhelming the metadata service. | 1000 |
| gcOverreplicatedLedgerWaitTime | How long the interval to trigger next garbage collection of overreplicated ledgers, in milliseconds. This should not be run very frequently since we read the metadata for all the ledgers on the bookie from zk. | 86400000 |
| gcOverreplicatedLedgerMaxConcurrentRequests | Max number of concurrent requests in garbage collection of overreplicated ledgers. | 1000 |
| isForceGCAllowWhenNoSpace | Whether force compaction is allowed when the disk is full or almost full. Forcing GC may get some space back, but may also fill up disk space more quickly. This is because new log files are created before GC, while old garbage log files are deleted after GC. | false |
(bookkeeper) 03/05: Only stop Gc for the fill disk for DbLedgerStorage (#4661)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 67eacd0e570e3d490b2a1ee54e36a8b2e86dbe1d
Author: Xiangying Meng <55...@users.noreply.github.com>
AuthorDate: Wed Sep 3 10:21:28 2025 +0800
Only stop Gc for the fill disk for DbLedgerStorage (#4661)
* Only stop Gc for the fill disk for DbLedgerStorage
(cherry picked from commit 0621ae6fce446c00f5becf0c81744bd8f9ec1c90)
---
.../bookie/storage/ldb/DbLedgerStorage.java | 2 +-
.../ldb/SingleDirectoryDbLedgerStorage.java | 40 +++++-
.../bookie/BookieStorageThresholdTest.java | 135 +++++++++++++++++++++
3 files changed, 171 insertions(+), 6 deletions(-)
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
index 60f752e226..e07a06b69c 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
@@ -449,7 +449,7 @@ public class DbLedgerStorage implements LedgerStorage {
}
@VisibleForTesting
- List<SingleDirectoryDbLedgerStorage> getLedgerStorageList() {
+ public List<SingleDirectoryDbLedgerStorage> getLedgerStorageList() {
return ledgerStorageList;
}
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
index eb1372bc3a..1422743df5 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java
@@ -39,6 +39,7 @@ import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.PrimitiveIterator.OfLong;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ExecutorService;
@@ -149,6 +150,8 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
private final Counter flushExecutorTime;
private final boolean singleLedgerDirs;
+ private final String ledgerBaseDir;
+ private final String indexBaseDir;
public SingleDirectoryDbLedgerStorage(ServerConfiguration conf, LedgerManager ledgerManager,
LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager,
@@ -158,8 +161,7 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
throws IOException {
checkArgument(ledgerDirsManager.getAllLedgerDirs().size() == 1,
"Db implementation only allows for one storage dir");
-
- String ledgerBaseDir = ledgerDirsManager.getAllLedgerDirs().get(0).getPath();
+ ledgerBaseDir = ledgerDirsManager.getAllLedgerDirs().get(0).getPath();
// indexBaseDir default use ledgerBaseDir
String indexBaseDir = ledgerBaseDir;
if (CollectionUtils.isEmpty(indexDirsManager.getAllLedgerDirs())
@@ -172,6 +174,7 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
log.info("indexDir is specified a separate dir, creating single directory db ledger storage on {}",
indexBaseDir);
}
+ this.indexBaseDir = indexBaseDir;
StatsLogger ledgerIndexDirStatsLogger = statsLogger
.scopeLabel("ledgerDir", ledgerBaseDir)
@@ -228,9 +231,9 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
flushExecutorTime.addLatency(0, TimeUnit.NANOSECONDS);
});
- ledgerDirsManager.addLedgerDirsListener(getLedgerDirsListener());
+ ledgerDirsManager.addLedgerDirsListener(getLedgerDirsListener(ledgerBaseDir));
if (!ledgerBaseDir.equals(indexBaseDir)) {
- indexDirsManager.addLedgerDirsListener(getLedgerDirsListener());
+ indexDirsManager.addLedgerDirsListener(getLedgerDirsListener(indexBaseDir));
}
}
@@ -1148,11 +1151,19 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
"getListOfEntriesOfLedger method is currently unsupported for SingleDirectoryDbLedgerStorage");
}
- private LedgerDirsManager.LedgerDirsListener getLedgerDirsListener() {
+ private LedgerDirsManager.LedgerDirsListener getLedgerDirsListener(String diskPath) {
return new LedgerDirsListener() {
+ private final String currentFilePath = diskPath;
+
+ private boolean isCurrentFile(File disk) {
+ return Objects.equals(disk.getPath(), currentFilePath);
+ }
@Override
public void diskAlmostFull(File disk) {
+ if (!isCurrentFile(disk)) {
+ return;
+ }
if (gcThread.isForceGCAllowWhenNoSpace()) {
gcThread.enableForceGC();
} else {
@@ -1162,6 +1173,9 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
@Override
public void diskFull(File disk) {
+ if (!isCurrentFile(disk)) {
+ return;
+ }
if (gcThread.isForceGCAllowWhenNoSpace()) {
gcThread.enableForceGC();
} else {
@@ -1182,6 +1196,9 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
@Override
public void diskWritable(File disk) {
+ if (!isCurrentFile(disk)) {
+ return;
+ }
// we have enough space now
if (gcThread.isForceGCAllowWhenNoSpace()) {
// disable force gc.
@@ -1195,6 +1212,9 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
@Override
public void diskJustWritable(File disk) {
+ if (!isCurrentFile(disk)) {
+ return;
+ }
if (gcThread.isForceGCAllowWhenNoSpace()) {
// if a disk is just writable, we still need force gc.
gcThread.enableForceGC();
@@ -1297,4 +1317,14 @@ public class SingleDirectoryDbLedgerStorage implements CompactableLedgerStorage
DbLedgerStorageStats getDbLedgerStorageStats() {
return dbLedgerStorageStats;
}
+
+ @VisibleForTesting
+ public String getLedgerBaseDir() {
+ return ledgerBaseDir;
+ }
+
+ @VisibleForTesting
+ public String getIndexBaseDir() {
+ return indexBaseDir;
+ }
}
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieStorageThresholdTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieStorageThresholdTest.java
index b242a0d866..d322c99d87 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieStorageThresholdTest.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieStorageThresholdTest.java
@@ -21,21 +21,28 @@
package org.apache.bookkeeper.bookie;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.io.File;
import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.bookkeeper.bookie.LedgerDirsManager.LedgerDirsListener;
import org.apache.bookkeeper.bookie.LedgerDirsManager.NoWritableLedgerDirException;
+import org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage;
import org.apache.bookkeeper.client.BookKeeper.DigestType;
import org.apache.bookkeeper.client.LedgerHandle;
import org.apache.bookkeeper.common.testing.annotations.FlakyTest;
import org.apache.bookkeeper.conf.ServerConfiguration;
+import org.apache.bookkeeper.conf.TestBKConfiguration;
import org.apache.bookkeeper.proto.BookieServer;
import org.apache.bookkeeper.test.BookKeeperClusterTestCase;
import org.apache.bookkeeper.util.DiskChecker;
@@ -250,4 +257,132 @@ public class BookieStorageThresholdTest extends BookKeeperClusterTestCase {
Thread.sleep(500);
assertFalse("Bookie should be transitioned to ReadWrite", bookie.isReadOnly());
}
+
+ @org.junit.Test
+ public void testStopGCOnCorrespondingDiskWhenDiskFull() throws Exception {
+ // 1. Create test directories
+ File ledgerDir1 = tmpDirs.createNew("ledger", "test1");
+ File ledgerDir2 = tmpDirs.createNew("ledger", "test2");
+
+ // 2. Configure Bookie
+ ServerConfiguration conf = TestBKConfiguration.newServerConfiguration();
+ conf.setGcWaitTime(1000);
+ conf.setLedgerDirNames(new String[] { ledgerDir1.getPath(), ledgerDir2.getPath() });
+ conf.setDiskCheckInterval(100); // Shorten disk check interval
+ conf.setLedgerStorageClass(DbLedgerStorage.class.getName());
+
+ // 3. Start Bookie and obtain internal components
+ Bookie bookie = new TestBookieImpl(conf);
+ BookieImpl bookieImpl = (BookieImpl) bookie;
+ LedgerDirsManager ledgerDirsManager = bookieImpl.getLedgerDirsManager();
+
+ // 4. Create custom disk checker (key step)
+ GCTestDiskChecker diskChecker = new GCTestDiskChecker(
+ conf.getDiskUsageThreshold(),
+ conf.getDiskUsageWarnThreshold()
+ );
+ // Set directory status: dir1 full (100%), dir2 normal (50%)
+ File[] currentDirectories = BookieImpl.getCurrentDirectories(new File[] { ledgerDir1, ledgerDir2 });
+ diskChecker.setUsageMap(currentDirectories[0], 1.0f); // 100% usage
+ diskChecker.setUsageMap(currentDirectories[1], 0.5f); // 50% usage
+
+ // 5. Replace Bookie's disk checker
+ bookieImpl.dirsMonitor.shutdown(); // Stop default monitor
+ bookieImpl.dirsMonitor = new LedgerDirsMonitor(
+ conf,
+ diskChecker,
+ Collections.singletonList(ledgerDirsManager)
+ );
+ bookieImpl.dirsMonitor.start();
+
+ // 6. Add disk state listener
+ CountDownLatch dir1Full = new CountDownLatch(1);
+ CountDownLatch dir1Writable = new CountDownLatch(1);
+
+ ledgerDirsManager.addLedgerDirsListener(new LedgerDirsListener() {
+ @Override
+ public void diskFull(File disk) {
+ if (disk.equals(currentDirectories[0])) {
+ dir1Full.countDown();
+ }
+ }
+
+ @Override
+ public void diskWritable(File disk) {
+ if (disk.equals(currentDirectories[0])) {
+ dir1Writable.countDown();
+ }
+ }
+ });
+
+ // 7. Wait for state update (ensure event is triggered)
+ assertTrue("dir1 did not trigger full state", dir1Full.await(30, TimeUnit.SECONDS));
+
+ // 8. Verify directory status
+ List<File> fullDirs = ledgerDirsManager.getFullFilledLedgerDirs();
+ List<File> writableDirs = ledgerDirsManager.getWritableLedgerDirs();
+
+ assertTrue("dir1 should be marked as full", fullDirs.contains(currentDirectories[0]));
+ assertTrue("dir2 should remain writable", writableDirs.contains(currentDirectories[1]));
+ assertEquals("Only 1 writable directory should remain", 1, writableDirs.size());
+
+ // 9. Verify GC status
+ ((DbLedgerStorage) bookieImpl.ledgerStorage).getLedgerStorageList().forEach(storage -> {
+ if (Objects.equals(storage.getLedgerBaseDir(), currentDirectories[0].getPath())) {
+ assertTrue("dir1 should suspend minor GC", storage.isMinorGcSuspended());
+ assertTrue("dir1 should suspend major GC", storage.isMajorGcSuspended());
+ } else {
+ assertFalse("dir2 should not suspend minor GC", storage.isMinorGcSuspended());
+ assertFalse("dir2 should not suspend major GC", storage.isMajorGcSuspended());
+ }
+ });
+
+ // 10. Restore dir1 status
+ diskChecker.setUsageMap(currentDirectories[0], 0.5f); // 50% usage
+ assertTrue("dir1 did not become writable again", dir1Writable.await(3, TimeUnit.SECONDS));
+
+ // 11. Verify GC status after recovery
+ ((DbLedgerStorage) bookieImpl.ledgerStorage).getLedgerStorageList().forEach(storage -> {
+ if (Objects.equals(storage.getLedgerBaseDir(), currentDirectories[0].getPath())) {
+ assertFalse("dir1 should not suspend minor GC", storage.isMinorGcSuspended());
+ assertFalse("dir1 should not suspend major GC", storage.isMajorGcSuspended());
+ } else {
+ assertFalse("dir2 should not suspend minor GC", storage.isMinorGcSuspended());
+ assertFalse("dir2 should not suspend major GC", storage.isMajorGcSuspended());
+ }
+ });
+
+ // 12. Cleanup
+ bookie.shutdown();
+ }
+
+ // Custom disk checker (simulate different usage for directories)
+ static class GCTestDiskChecker extends DiskChecker {
+ private final Map<File, Float> usageMap = new ConcurrentHashMap<>();
+
+ public GCTestDiskChecker(float threshold, float warnThreshold) {
+ super(threshold, warnThreshold);
+ }
+
+ // Set simulated usage for a directory
+ public void setUsageMap(File dir, float usage) {
+ usageMap.put(dir, usage);
+ }
+
+ @Override
+ public float checkDir(File dir) throws DiskErrorException, DiskWarnThresholdException, DiskOutOfSpaceException {
+ Float usage = usageMap.get(dir);
+ if (usage == null) {
+ return super.checkDir(dir); // Default behavior
+ }
+ // Throw exception based on preset usage rate
+ if (usage >= 1.0) {
+ throw new DiskOutOfSpaceException("Simulated disk full", usage);
+ } else if (usage >= 0.9) {
+ throw new DiskWarnThresholdException("Simulated disk warning", usage);
+ }
+ return usage;
+ }
+ }
+
}
(bookkeeper) 01/05: [fix]compile error of the file native_io_jni.c for the env jdk11 & windows (#4665)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.17
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 22fe67ffd2e14c62e8137fc3d3f4e627bdfdc0c7
Author: fengyubiao <yu...@streamnative.io>
AuthorDate: Wed Oct 22 12:53:14 2025 +0800
[fix]compile error of the file native_io_jni.c for the env jdk11 & windows (#4665)
Co-authored-by: fengyubiao <fe...@100tal.com>
(cherry picked from commit 5154149936e3f8187ae9e2755f43ea9c0ecb866e)
---
native-io/src/main/native-io-jni/cpp/native_io_jni.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/native-io/src/main/native-io-jni/cpp/native_io_jni.c b/native-io/src/main/native-io-jni/cpp/native_io_jni.c
index d3bc164bec..7be468518b 100644
--- a/native-io/src/main/native-io-jni/cpp/native_io_jni.c
+++ b/native-io/src/main/native-io-jni/cpp/native_io_jni.c
@@ -20,11 +20,16 @@
*/
#define _GNU_SOURCE
+#include <jni.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
+#ifdef _WIN32
+#include <io.h>
+#else
#include <unistd.h>
+#endif
#include <org_apache_bookkeeper_common_util_nativeio_NativeIOJni.h>
@@ -165,7 +170,14 @@ JNIEXPORT jint JNICALL
Java_org_apache_bookkeeper_common_util_nativeio_NativeIOJni_fsync(JNIEnv * env,
jclass clazz,
jint fd) {
- int res = fsync(fd);
+ int res;
+
+ // Guarantee compatibility for winsows.
+ #ifdef _WIN32
+ res = _commit((int)fd);
+ #else
+ res = fsync((int)fd);
+ #endif
if (res == -1) {
throwExceptionWithErrno(env, "Failed to fsync");