You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2023/05/18 23:48:06 UTC
[GitHub] [bookkeeper] merlimat opened a new pull request, #3959: Avoid compaction to trigger extra flushes DbLedgerStorage
merlimat opened a new pull request, #3959:
URL: https://github.com/apache/bookkeeper/pull/3959
### Motivation
This PR solves the problem described #3940.
Because compaction is triggering a flush after each entry log file is compacted, the flush on the DbLedgerStorage causes the write cache to be underutilized and ultimately it will slow down the writes.
### Changes
The call to `flush()` in `updateEntriesLocations()` was there to ensure that every entry log file we might be compacting has the indexes already flushed into RocksDB. If that's not the case, the compaction might first write the new location and the regular flush will override it with the old location, to a file that is already deleted.
In practice, we don't actually need to trigger a new flush for that to be valid, we just need to ensure that if there is a pending `flush()`, we wait until it's done. If there are no pending flushes, we're also good to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3959: Avoid compaction to trigger extra flushes DbLedgerStorage
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on PR #3959:
URL: https://github.com/apache/bookkeeper/pull/3959#issuecomment-1556376529
@merlimat There is one test that failed related to compaction, please help take a look, thanks.
```
Error: Tests run: 14, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.967 s <<< FAILURE! - in org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest
Error: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest.testBookieCompaction Time elapsed: 0.185 s <<< ERROR!
java.io.IOException: Invalid entry size(file=/tmp/bkTest9941833862601936794.dir/current/0.log,offset=8196,maxSaneEntrySize=5[242](https://github.com/apache/bookkeeper/actions/runs/5026782107/jobs/9015647162?pr=3959#step:8:243)380,readEntrySize=0)
at org.apache.bookkeeper.bookie.storage.directentrylogger.DirectReader.readEntryAt(DirectReader.java:190)
at org.apache.bookkeeper.bookie.storage.directentrylogger.DirectEntryLogger.internalReadEntry(DirectEntryLogger.java:249)
at org.apache.bookkeeper.bookie.storage.directentrylogger.DirectEntryLogger.readEntry(DirectEntryLogger.java:211)
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.doGetEntry(SingleDirectoryDbLedgerStorage.java:636)
at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.getEntry(SingleDirectoryDbLedgerStorage.java:558)
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.getEntry(DbLedgerStorage.java:361)
at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testBookieCompaction(DbLedgerStorageTest.java:[243](https://github.com/apache/bookkeeper/actions/runs/5026782107/jobs/9015647162?pr=3959#step:8:244))
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat merged pull request #3959: Avoid compaction to trigger extra flushes DbLedgerStorage
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #3959:
URL: https://github.com/apache/bookkeeper/pull/3959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org