You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/06/26 14:40:07 UTC
[GitHub] [bookkeeper] hangc0276 opened a new pull request, #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
hangc0276 opened a new pull request, #3366:
URL: https://github.com/apache/bookkeeper/pull/3366
### Motivation
Task 7 of BP-47, DbLedgerStorage add direct entry logger support.
This is the last Pr of BP-47
### Modification
Add parameter `dbStorage_directIOEntryLogger` to control whether use direct entrylogger for DbLedgerStorage or not. Default value is false.
Others
The commit is made by @ivankelly . This is the sub-task of https://github.com/apache/bookkeeper/pull/2932 , which was pushed out by @mauricebarnum. However, this PR contains too many changes and the number of code lines reaches 8K+, and it is hard to review. According to your suggestion https://github.com/apache/bookkeeper/pull/2932#issuecomment-1072882245, and after communicating with @mauricebarnum, we are planning to divide the PR into 6 PRs, Please refer to https://github.com/apache/bookkeeper/issues/2943#issuecomment-1086446251.
However, @mauricebarnum doesn't have enough time to deal with those issues, and we desperately need this feature. After communicating with @mauricebarnum and @merlimat , I help to work on dividing the PRs and pushing them out. We are hoping to contain this feature in BookKeeper 4.16.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1169476449
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on a diff in pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#discussion_r910599379
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorageWithDirectEntryLoggerTest.java:
##########
@@ -0,0 +1,36 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.bookie.storage.ldb;
+
+import org.junit.Before;
+
+/**
+ * Unit test for {@link DbLedgerStorage} with directIO entrylogger.
+ */
+public class DbLedgerStorageWithDirectEntryLoggerTest extends DbLedgerStorageTest {
+
+ @Override
+ @Before
+ public void setup() throws Exception {
+ super.setup();
+ conf.setProperty("dbStorage_directIOEntryLogger", true);
Review Comment:
Is it possible to add some assertions in the tests about the fact that we really loaded this implementation?
The tests may pass because we aren't using the new implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1169519328
Ping @merlimat @eolivelli @ivankelly @Vanlightly Please help take a look, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1171158271
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1171216997
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 commented on a diff in pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
dlg99 commented on code in PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#discussion_r908657094
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java:
##########
@@ -91,6 +106,16 @@ public class DbLedgerStorage implements LedgerStorage {
static final String READ_AHEAD_CACHE_BATCH_SIZE = "dbStorage_readAheadCacheBatchSize";
private static final int DEFAULT_READ_AHEAD_CACHE_BATCH_SIZE = 100;
+ private static final long DEFAULT_DIRECT_IO_TOTAL_WRITEBUFFER_SIZE_MB =
+ (long) (0.125 * PlatformDependent.maxDirectMemory())
Review Comment:
PlatformDependent.estimateMaxDirectMemory()
See https://github.com/apache/bookkeeper/pull/2989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1171246471
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1166769377
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1171114689
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli merged pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
eolivelli merged PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1166685582
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#discussion_r912317716
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorageWithDirectEntryLoggerTest.java:
##########
@@ -0,0 +1,36 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.bookie.storage.ldb;
+
+import org.junit.Before;
+
+/**
+ * Unit test for {@link DbLedgerStorage} with directIO entrylogger.
+ */
+public class DbLedgerStorageWithDirectEntryLoggerTest extends DbLedgerStorageTest {
+
+ @Override
+ @Before
+ public void setup() throws Exception {
+ super.setup();
+ conf.setProperty("dbStorage_directIOEntryLogger", true);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3366: BP-47 (task7): DbLedgerStorage add direct entry logger support
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3366:
URL: https://github.com/apache/bookkeeper/pull/3366#issuecomment-1168087991
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org