You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by yo...@apache.org on 2022/09/21 11:09:53 UTC
[bookkeeper] branch branch-4.15 updated (f0182edbfb -> e10aa34416)
This is an automated email from the ASF dual-hosted git repository.
yong pushed a change to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
from f0182edbfb AutoRecovery - Do not call shutdown() on the main ZookKeeper client thread (#3487)
new 58db136bf7 Add latency stats for entry location index lookup so that possible RocksDB bottleneck can be detected (#3444)
new e84b08c4d6 Optimize log for failed to write entry (#3463)
new 32151b1602 Bump snakeyaml from 1.30 to 1.31 to solve CVE-2022-25857 (#3469)
new b876ebc353 Fix the deadlock when only using io thread to handle request (#3480)
new fb8cf1e3e5 upgrade hadoop version to 3.2.4 (#3485)
new 3b556e8fc6 Check if channel closed before processing read request (#3486)
new e10aa34416 Bump snakeyaml from 1.31 to 1.32 to solve CVE-2022-38752 (#3491)
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/bookkeeper/test/TestStatsProvider.java | 4 ++
.../bookie/storage/ldb/EntryLocationIndex.java | 12 +++-
.../storage/ldb/EntryLocationIndexStats.java | 9 +++
.../org/apache/bookkeeper/client/PendingAddOp.java | 4 +-
.../bookkeeper/proto/PacketProcessorBase.java | 8 ++-
.../bookkeeper/proto/ReadEntryProcessor.java | 7 ++
.../bookkeeper/proto/ReadEntryProcessorV3.java | 7 ++
.../bookie/storage/ldb/EntryLocationIndexTest.java | 29 ++++++++
.../client/BookieRecoveryUseIOThreadTest.java | 77 ++++++++++++++++++++++
.../proto/ForceLedgerProcessorV3Test.java | 1 +
.../proto/LongPollReadEntryProcessorV3Test.java | 1 +
.../bookkeeper/proto/ReadEntryProcessorTest.java | 1 +
.../bookkeeper/proto/WriteEntryProcessorTest.java | 1 +
.../proto/WriteEntryProcessorV3Test.java | 1 +
.../org/apache/bookkeeper/stats/OpStatsLogger.java | 8 +--
pom.xml | 4 +-
16 files changed, 162 insertions(+), 12 deletions(-)
create mode 100644 bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryUseIOThreadTest.java
[bookkeeper] 03/07: Bump snakeyaml from 1.30 to 1.31 to solve CVE-2022-25857 (#3469)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 32151b1602d2e7b545c9c0e00e013fd502d6a4c9
Author: ZhangJian He <sh...@gmail.com>
AuthorDate: Wed Sep 7 07:50:35 2022 +0800
Bump snakeyaml from 1.30 to 1.31 to solve CVE-2022-25857 (#3469)
(cherry picked from commit 457eaf8dbc0101deef2159e7ec1c312ab66af2db)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index cb20f5198d..ba20c69859 100644
--- a/pom.xml
+++ b/pom.xml
@@ -171,7 +171,7 @@
<rocksdb.version>6.29.4.1</rocksdb.version>
<shrinkwrap.version>3.0.1</shrinkwrap.version>
<slf4j.version>1.7.32</slf4j.version>
- <snakeyaml.version>1.30</snakeyaml.version>
+ <snakeyaml.version>1.31</snakeyaml.version>
<spotbugs-annotations.version>4.6.0</spotbugs-annotations.version>
<javax-annotations-api.version>1.3.2</javax-annotations-api.version>
<testcontainers.version>1.15.1</testcontainers.version>
[bookkeeper] 07/07: Bump snakeyaml from 1.31 to 1.32 to solve CVE-2022-38752 (#3491)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit e10aa3441617ed56a00c576616469a2993f33518
Author: Tian Luo <fu...@outlook.com>
AuthorDate: Fri Sep 16 08:49:47 2022 +0800
Bump snakeyaml from 1.31 to 1.32 to solve CVE-2022-38752 (#3491)
(cherry picked from commit fc85ce0bbe9e1e33a6e92a6eebab432eadb4e34e)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 1252c2c11e..d99ca4603d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -171,7 +171,7 @@
<rocksdb.version>6.29.4.1</rocksdb.version>
<shrinkwrap.version>3.0.1</shrinkwrap.version>
<slf4j.version>1.7.32</slf4j.version>
- <snakeyaml.version>1.31</snakeyaml.version>
+ <snakeyaml.version>1.32</snakeyaml.version>
<spotbugs-annotations.version>4.6.0</spotbugs-annotations.version>
<javax-annotations-api.version>1.3.2</javax-annotations-api.version>
<testcontainers.version>1.15.1</testcontainers.version>
[bookkeeper] 06/07: Check if channel closed before processing read request (#3486)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 3b556e8fc6efce81f8cfdfe18389962ae28754c8
Author: Michael Marshall <mm...@apache.org>
AuthorDate: Mon Sep 19 02:57:51 2022 -0700
Check if channel closed before processing read request (#3486)
* Check if channel closed before processing read request
* Add missed call to onReadRequestFinish()
* Fix test
* Mock more tests
Co-authored-by: Nicolò Boschi <bo...@gmail.com>
(cherry picked from commit 1313b8e2e7964a0bbf2e221f49fe30a1fd812d31)
---
.../main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java | 7 +++++++
.../java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java | 7 +++++++
.../org/apache/bookkeeper/proto/ForceLedgerProcessorV3Test.java | 1 +
.../apache/bookkeeper/proto/LongPollReadEntryProcessorV3Test.java | 1 +
.../java/org/apache/bookkeeper/proto/ReadEntryProcessorTest.java | 1 +
.../java/org/apache/bookkeeper/proto/WriteEntryProcessorTest.java | 1 +
.../org/apache/bookkeeper/proto/WriteEntryProcessorV3Test.java | 1 +
7 files changed, 19 insertions(+)
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java
index 7c8c2c8493..3373bf8ffd 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java
@@ -61,6 +61,13 @@ class ReadEntryProcessor extends PacketProcessorBase<ReadRequest> {
if (LOG.isDebugEnabled()) {
LOG.debug("Received new read request: {}", request);
}
+ if (!channel.isOpen()) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Dropping read request for closed channel: {}", channel);
+ }
+ requestProcessor.onReadRequestFinish();
+ return;
+ }
int errorCode = BookieProtocol.EOK;
long startTimeNanos = MathUtils.nowInNano();
ByteBuf data = null;
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java
index e03cb35fa5..75a10df2ae 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java
@@ -249,6 +249,13 @@ class ReadEntryProcessorV3 extends PacketProcessorBaseV3 {
public void safeRun() {
requestProcessor.getRequestStats().getReadEntrySchedulingDelayStats().registerSuccessfulEvent(
MathUtils.elapsedNanos(enqueueNanos), TimeUnit.NANOSECONDS);
+ if (!channel.isOpen()) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Dropping read request for closed channel: {}", channel);
+ }
+ requestProcessor.onReadRequestFinish();
+ return;
+ }
if (!isVersionCompatible()) {
ReadResponse readResponse = ReadResponse.newBuilder()
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ForceLedgerProcessorV3Test.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ForceLedgerProcessorV3Test.java
index 6c6eea7adb..54460770f1 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ForceLedgerProcessorV3Test.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ForceLedgerProcessorV3Test.java
@@ -72,6 +72,7 @@ public class ForceLedgerProcessorV3Test {
.build())
.build();
channel = mock(Channel.class);
+ when(channel.isOpen()).thenReturn(true);
bookie = mock(Bookie.class);
requestProcessor = mock(BookieRequestProcessor.class);
when(requestProcessor.getBookie()).thenReturn(bookie);
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/LongPollReadEntryProcessorV3Test.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/LongPollReadEntryProcessorV3Test.java
index 393d5ddf33..9eae1b9c0d 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/LongPollReadEntryProcessorV3Test.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/LongPollReadEntryProcessorV3Test.java
@@ -83,6 +83,7 @@ public class LongPollReadEntryProcessorV3Test {
.build();
Channel channel = mock(Channel.class);
+ when(channel.isOpen()).thenReturn(true);
Bookie bookie = mock(Bookie.class);
BookieRequestProcessor requestProcessor = mock(BookieRequestProcessor.class);
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ReadEntryProcessorTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ReadEntryProcessorTest.java
index 4d18645b90..114d614b28 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ReadEntryProcessorTest.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/ReadEntryProcessorTest.java
@@ -58,6 +58,7 @@ public class ReadEntryProcessorTest {
@Before
public void setup() throws IOException, BookieException {
channel = mock(Channel.class);
+ when(channel.isOpen()).thenReturn(true);
bookie = mock(Bookie.class);
requestProcessor = mock(BookieRequestProcessor.class);
when(requestProcessor.getBookie()).thenReturn(bookie);
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorTest.java
index e2c141b741..150e0c3089 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorTest.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorTest.java
@@ -67,6 +67,7 @@ public class WriteEntryProcessorTest {
new byte[0],
Unpooled.wrappedBuffer("test-entry-data".getBytes(UTF_8)));
channel = mock(Channel.class);
+ when(channel.isOpen()).thenReturn(true);
bookie = mock(Bookie.class);
requestProcessor = mock(BookieRequestProcessor.class);
when(requestProcessor.getBookie()).thenReturn(bookie);
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorV3Test.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorV3Test.java
index 3c81d73a5e..2175b26a5b 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorV3Test.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/WriteEntryProcessorV3Test.java
@@ -77,6 +77,7 @@ public class WriteEntryProcessorV3Test {
.build())
.build();
channel = mock(Channel.class);
+ when(channel.isOpen()).thenReturn(true);
bookie = mock(Bookie.class);
requestProcessor = mock(BookieRequestProcessor.class);
when(requestProcessor.getBookie()).thenReturn(bookie);
[bookkeeper] 01/07: Add latency stats for entry location index lookup so that possible RocksDB bottleneck can be detected (#3444)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit 58db136bf7856e39120947fe431ae16c7d2ef1fa
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Tue Aug 16 13:46:33 2022 +0300
Add latency stats for entry location index lookup so that possible RocksDB bottleneck can be detected (#3444)
* Add operation latency stats for entry location lookup
- this metric will help detecting when the bottleneck is in the entry location index lookup operations
and RocksDB tuning is needed
* Address review feedback: fix issue with eventLatencyMillis variable
* Rename misleading parameter name
(cherry picked from commit 19fd8f74265130f76f55b04a24cc0ee5c7a634a5)
---
.../apache/bookkeeper/test/TestStatsProvider.java | 4 +++
.../bookie/storage/ldb/EntryLocationIndex.java | 12 ++++++++-
.../storage/ldb/EntryLocationIndexStats.java | 9 +++++++
.../bookie/storage/ldb/EntryLocationIndexTest.java | 29 ++++++++++++++++++++++
.../org/apache/bookkeeper/stats/OpStatsLogger.java | 8 +++---
5 files changed, 57 insertions(+), 5 deletions(-)
diff --git a/bookkeeper-common/src/test/java/org/apache/bookkeeper/test/TestStatsProvider.java b/bookkeeper-common/src/test/java/org/apache/bookkeeper/test/TestStatsProvider.java
index f486075651..c2a06d6f41 100644
--- a/bookkeeper-common/src/test/java/org/apache/bookkeeper/test/TestStatsProvider.java
+++ b/bookkeeper-common/src/test/java/org/apache/bookkeeper/test/TestStatsProvider.java
@@ -146,6 +146,10 @@ public class TestStatsProvider implements StatsProvider {
public synchronized long getSuccessCount() {
return successCount;
}
+
+ public synchronized long getFailureCount() {
+ return failureCount;
+ }
}
/**
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndex.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndex.java
index 468f268aab..dc8e947ffc 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndex.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndex.java
@@ -34,6 +34,7 @@ import org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorage.Batch;
import org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageFactory.DbConfigType;
import org.apache.bookkeeper.conf.ServerConfiguration;
import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.util.MathUtils;
import org.apache.bookkeeper.util.collections.ConcurrentLongHashSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -75,6 +76,8 @@ public class EntryLocationIndex implements Closeable {
LongPairWrapper key = LongPairWrapper.get(ledgerId, entryId);
LongWrapper value = LongWrapper.get();
+ long startTimeNanos = MathUtils.nowInNano();
+ boolean operationSuccess = false;
try {
if (locationsDb.get(key.array, value.array) < 0) {
if (log.isDebugEnabled()) {
@@ -82,11 +85,18 @@ public class EntryLocationIndex implements Closeable {
}
return 0;
}
-
+ operationSuccess = true;
return value.getValue();
} finally {
key.recycle();
value.recycle();
+ if (operationSuccess) {
+ stats.getLookupEntryLocationStats()
+ .registerSuccessfulEvent(MathUtils.elapsedNanos(startTimeNanos), TimeUnit.NANOSECONDS);
+ } else {
+ stats.getLookupEntryLocationStats()
+ .registerFailedEvent(MathUtils.elapsedNanos(startTimeNanos), TimeUnit.NANOSECONDS);
+ }
}
}
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexStats.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexStats.java
index dd87f7b92b..80bdda4825 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexStats.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexStats.java
@@ -25,6 +25,7 @@ import static org.apache.bookkeeper.bookie.BookKeeperServerStats.CATEGORY_SERVER
import java.util.function.Supplier;
import lombok.Getter;
import org.apache.bookkeeper.stats.Gauge;
+import org.apache.bookkeeper.stats.OpStatsLogger;
import org.apache.bookkeeper.stats.StatsLogger;
import org.apache.bookkeeper.stats.annotations.StatsDoc;
@@ -40,6 +41,7 @@ import org.apache.bookkeeper.stats.annotations.StatsDoc;
class EntryLocationIndexStats {
private static final String ENTRIES_COUNT = "entries-count";
+ private static final String LOOKUP_ENTRY_LOCATION = "lookup-entry-location";
@StatsDoc(
name = ENTRIES_COUNT,
@@ -47,6 +49,12 @@ class EntryLocationIndexStats {
)
private final Gauge<Long> entriesCountGauge;
+ @StatsDoc(
+ name = LOOKUP_ENTRY_LOCATION,
+ help = "operation stats of looking up entry location"
+ )
+ private final OpStatsLogger lookupEntryLocationStats;
+
EntryLocationIndexStats(StatsLogger statsLogger,
Supplier<Long> entriesCountSupplier) {
entriesCountGauge = new Gauge<Long>() {
@@ -61,6 +69,7 @@ class EntryLocationIndexStats {
}
};
statsLogger.registerGauge(ENTRIES_COUNT, entriesCountGauge);
+ lookupEntryLocationStats = statsLogger.getOpStatsLogger(LOOKUP_ENTRY_LOCATION);
}
}
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexTest.java
index f1cefed161..c9f505c5f5 100644
--- a/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexTest.java
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/EntryLocationIndexTest.java
@@ -21,12 +21,14 @@
package org.apache.bookkeeper.bookie.storage.ldb;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
import java.io.File;
import java.io.IOException;
import org.apache.bookkeeper.conf.ServerConfiguration;
import org.apache.bookkeeper.stats.NullStatsLogger;
+import org.apache.bookkeeper.test.TestStatsProvider;
import org.junit.Test;
/**
@@ -140,4 +142,31 @@ public class EntryLocationIndexTest {
idx.removeOffsetFromDeletedLedgers();
assertEquals(0, idx.getLocation(40312, 10));
}
+
+ @Test
+ public void testEntryIndexLookupLatencyStats() throws IOException {
+ File tmpDir = File.createTempFile("bkTest", ".dir");
+ tmpDir.delete();
+ tmpDir.mkdir();
+ tmpDir.deleteOnExit();
+
+ TestStatsProvider statsProvider = new TestStatsProvider();
+ EntryLocationIndex idx = new EntryLocationIndex(serverConfiguration, KeyValueStorageRocksDB.factory,
+ tmpDir.getAbsolutePath(), statsProvider.getStatsLogger("scope"));
+
+ // Add some dummy indexes
+ idx.addLocation(40313, 11, 5);
+
+ // successful lookup
+ assertEquals(5, idx.getLocation(40313, 11));
+ TestStatsProvider.TestOpStatsLogger lookupEntryLocationOpStats =
+ statsProvider.getOpStatsLogger("scope.lookup-entry-location");
+ assertEquals(1, lookupEntryLocationOpStats.getSuccessCount());
+ assertTrue(lookupEntryLocationOpStats.getSuccessAverage() > 0);
+
+ // failed lookup
+ assertEquals(0, idx.getLocation(12345, 1));
+ assertEquals(1, lookupEntryLocationOpStats.getFailureCount());
+ assertEquals(1, lookupEntryLocationOpStats.getSuccessCount());
+ }
}
diff --git a/bookkeeper-stats/src/main/java/org/apache/bookkeeper/stats/OpStatsLogger.java b/bookkeeper-stats/src/main/java/org/apache/bookkeeper/stats/OpStatsLogger.java
index cef83042d7..3c9c730307 100644
--- a/bookkeeper-stats/src/main/java/org/apache/bookkeeper/stats/OpStatsLogger.java
+++ b/bookkeeper-stats/src/main/java/org/apache/bookkeeper/stats/OpStatsLogger.java
@@ -25,18 +25,18 @@ public interface OpStatsLogger {
/**
* Increment the failed op counter with the given eventLatency.
- * @param eventLatencyMillis The event latency
+ * @param eventLatency The event latency
* @param unit
*/
- void registerFailedEvent(long eventLatencyMillis, TimeUnit unit);
+ void registerFailedEvent(long eventLatency, TimeUnit unit);
/**
* An operation succeeded with the given eventLatency. Update
* stats to reflect the same
- * @param eventLatencyMillis The event latency
+ * @param eventLatency The event latency
* @param unit
*/
- void registerSuccessfulEvent(long eventLatencyMillis, TimeUnit unit);
+ void registerSuccessfulEvent(long eventLatency, TimeUnit unit);
/**
* An operation with the given value succeeded.
[bookkeeper] 02/07: Optimize log for failed to write entry (#3463)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit e84b08c4d6024027471eba829e72881584b7e59e
Author: Xiaoyu Hou <An...@gmail.com>
AuthorDate: Mon Sep 5 10:55:43 2022 +0800
Optimize log for failed to write entry (#3463)
Descriptions of the changes in this PR:
Print bookie address when writing entry failed
### Motivation
Printing bookie info will be help for solving online problem
### Changes
Print bookie address when writing entry failed
(cherry picked from commit 1adc0414bd96fd4eda67837d980e9df37471f631)
---
.../src/main/java/org/apache/bookkeeper/client/PendingAddOp.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java
index 223ce182f7..ac52de07b7 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java
@@ -373,8 +373,8 @@ class PendingAddOp extends SafeRunnable implements WriteCallback {
}
}
} else {
- LOG.warn("Failed to write entry ({}, {}): {}",
- ledgerId, entryId, BKException.getMessage(rc));
+ LOG.warn("Failed to write entry ({}, {}) to bookie ({}, {}): {}",
+ ledgerId, entryId, bookieIndex, addr, BKException.getMessage(rc));
lh.handleBookieFailure(ImmutableMap.of(bookieIndex, addr));
}
return;
[bookkeeper] 04/07: Fix the deadlock when only using io thread to handle request (#3480)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit b876ebc353e6d658ede2903e45b23bb0df7d4210
Author: Yong Zhang <zh...@gmail.com>
AuthorDate: Mon Sep 12 18:49:32 2022 +0800
Fix the deadlock when only using io thread to handle request (#3480)
* Fix the deadlock when only using io thread to handle request
---
*Motivation*
If user don't configure the ReadWorker thread pool, the reqeust will
process with io thread. We cannot call await() from an IO thread,
if the socket buffer is full, that blocking call would cause a deadlock.
*Modification*
- only wait the promise when the thread is not io thread
* Fix the style issue
(cherry picked from commit 70bbc3830f07691cbdb87fe03958196fafc0effb)
---
.../bookkeeper/proto/PacketProcessorBase.java | 8 ++-
.../client/BookieRecoveryUseIOThreadTest.java | 77 ++++++++++++++++++++++
2 files changed, 82 insertions(+), 3 deletions(-)
diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBase.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBase.java
index 7121579862..07954d746a 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBase.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBase.java
@@ -18,9 +18,8 @@
package org.apache.bookkeeper.proto;
import io.netty.channel.Channel;
-
+import io.netty.channel.ChannelFuture;
import java.util.concurrent.TimeUnit;
-
import org.apache.bookkeeper.proto.BookieProtocol.Request;
import org.apache.bookkeeper.stats.OpStatsLogger;
import org.apache.bookkeeper.util.MathUtils;
@@ -138,7 +137,10 @@ abstract class PacketProcessorBase<T extends Request> extends SafeRunnable {
*/
protected void sendResponseAndWait(int rc, Object response, OpStatsLogger statsLogger) {
try {
- channel.writeAndFlush(response).await();
+ ChannelFuture future = channel.writeAndFlush(response);
+ if (!channel.eventLoop().inEventLoop()) {
+ future.await();
+ }
} catch (InterruptedException e) {
return;
}
diff --git a/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryUseIOThreadTest.java b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryUseIOThreadTest.java
new file mode 100644
index 0000000000..fa1dba0cda
--- /dev/null
+++ b/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryUseIOThreadTest.java
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.client;
+
+import java.io.IOException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.bookkeeper.conf.ClientConfiguration;
+import org.apache.bookkeeper.test.BookKeeperClusterTestCase;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class BookieRecoveryUseIOThreadTest extends BookKeeperClusterTestCase {
+
+ public BookieRecoveryUseIOThreadTest() {
+ super(1);
+ }
+
+ @Override
+ public void setUp() throws Exception {
+ baseConf.setNumAddWorkerThreads(0);
+ baseConf.setNumReadWorkerThreads(0);
+ baseConf.setNumHighPriorityWorkerThreads(0);
+ super.setUp();
+ }
+
+ @Test
+ public void testRecoveryClosedLedger() throws BKException, IOException, InterruptedException {
+ // test the v2 protocol when using IO thread to handle the request
+ ClientConfiguration conf = new ClientConfiguration();
+ conf.setMetadataServiceUri(zkUtil.getMetadataServiceUri());
+ conf.setUseV2WireProtocol(true);
+ AtomicInteger finalRc = new AtomicInteger(Integer.MAX_VALUE);
+ CountDownLatch latch = new CountDownLatch(1);
+ try (BookKeeper bkc = new BookKeeper(conf)) {
+ bkc.asyncCreateLedger(1, 1, BookKeeper.DigestType.CRC32, "".getBytes(),
+ new AsyncCallback.CreateCallback() {
+ @Override
+ public void createComplete(int rc, LedgerHandle lh, Object ctx) {
+ lh.asyncAddEntry("hello".getBytes(), new AsyncCallback.AddCallback() {
+ @Override
+ public void addComplete(int rc, LedgerHandle lh, long entryId, Object ctx) {
+ if (rc == BKException.Code.OK) {
+ bkc.asyncOpenLedger(lh.ledgerId, BookKeeper.DigestType.CRC32, "".getBytes(),
+ new AsyncCallback.OpenCallback() {
+ @Override
+ public void openComplete(int rc, LedgerHandle lh, Object ctx) {
+ finalRc.set(rc);
+ latch.countDown();
+ }
+ }, null);
+ }
+ }
+ }, null);
+ }
+ }, null);
+ latch.await();
+ }
+ Assert.assertEquals(finalRc.get(), org.apache.bookkeeper.client.api.BKException.Code.OK);
+ }
+}
[bookkeeper] 05/07: upgrade hadoop version to 3.2.4 (#3485)
Posted by yo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yong pushed a commit to branch branch-4.15
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
commit fb8cf1e3e5de340625d7cfae830e220d176a580a
Author: Hang Chen <ch...@apache.org>
AuthorDate: Tue Sep 13 11:47:37 2022 +0800
upgrade hadoop version to 3.2.4 (#3485)
(cherry picked from commit aba568087c6dff99a343346107153ed1d979677b)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index ba20c69859..1252c2c11e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -138,7 +138,7 @@
<grpc.version>1.45.1</grpc.version>
<guava.version>31.0.1-jre</guava.version>
<kerby.version>1.1.1</kerby.version>
- <hadoop.version>2.10.0</hadoop.version>
+ <hadoop.version>3.2.4</hadoop.version>
<hamcrest.version>1.3</hamcrest.version>
<hdrhistogram.version>2.1.10</hdrhistogram.version>
<jackson.version>2.13.2.20220328</jackson.version>