You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/08/26 21:56:05 UTC
[GitHub] [bookkeeper] merlimat opened a new pull request, #3458: Speed up the rebuildinding of RocksDB index
merlimat opened a new pull request, #3458:
URL: https://github.com/apache/bookkeeper/pull/3458
### Motivation
When rebuilding RocksDB index, if we're inserting the entries one by one, we're becoming bottlenecked on the RocksDB insertion. Instead we should add items to the index in batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] StevenLuMT commented on pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458#issuecomment-1251175266
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458#issuecomment-1236463789
Hi @merlimat, Would you please help resolve the check style issue?
```
Error: /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/LocationsIndexRebuildOp.java:37:1: Extra separation in import group before 'org.apache.bookkeeper.bookie.BookieImpl' [ImportOrder]
Audit done.
[INFO] There is 1 error reported by Checkstyle 9.3 with buildtools/src/main/resources/bookkeeper/checkstyle.xml ruleset.
Error: src/main/java/org/apache/bookkeeper/bookie/storage/ldb/LocationsIndexRebuildOp.java:[37,1] (imports) ImportOrder: Extra separation in import group before 'org.apache.bookkeeper.bookie.BookieImpl'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] StevenLuMT commented on pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458#issuecomment-1251172393
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] StevenLuMT commented on a diff in pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on code in PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458#discussion_r956854789
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/LocationsIndexRebuildOp.java:
##########
@@ -31,6 +31,9 @@
import java.util.Date;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
Review Comment:
new checkstyle not pass, please clean line:36 newline
<img width="1382" alt="image" src="https://user-images.githubusercontent.com/42990025/187113881-cb0191bf-3ecd-403b-b23f-e5b94d024e84.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat merged pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by GitBox <gi...@apache.org>.
merlimat merged PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] zymap commented on pull request #3458: Speed up the rebuildinding of RocksDB index
Posted by "zymap (via GitHub)" <gi...@apache.org>.
zymap commented on PR #3458:
URL: https://github.com/apache/bookkeeper/pull/3458#issuecomment-1437783513
This PR will cause the test to fail on branch-4.15. Remove it from 4.15.4 release.
<details> <summary>Error log</summary>
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project bookkeeper-server: There are test failures.
[ERROR]
[ERROR] Please refer to /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server && /Volumes/work/devtools/jdk-11.0.16.jdk/Contents/Home/bin/java -Xmx2G -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetection.level=paranoid --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.nio.channels.spi=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.stream=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/sun.net.dns=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/sun.security.jca=ALL-UNNAMED --add-opens java.xml/jdk.xml.internal=ALL-UNNAMED -jar /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server/target/surefire/surefirebooter14408819248685557633.jar /Volumes/w
ork/github.com/apache/bookkeeper/bookkeeper-server/target/surefire 2023-02-21T09-39-53_120-jvmRun1 surefire2887148242694623677tmp surefire_014040627733256951610tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests:
[ERROR] org.apache.bookkeeper.bookie.storage.ldb.LocationsIndexRebuildTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server && /Volumes/work/devtools/jdk-11.0.16.jdk/Contents/Home/bin/java -Xmx2G -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetection.level=paranoid --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.nio.channels.spi=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.stream=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/sun.net.dns=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/sun.security.jca=ALL-UNNAMED --add-opens java.xml/jdk.xml.internal=ALL-UNNAMED -jar /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server/target/surefire/surefirebooter14408819248685557633.jar /Volumes/w
ork/github.com/apache/bookkeeper/bookkeeper-server/target/surefire 2023-02-21T09-39-53_120-jvmRun1 surefire2887148242694623677tmp surefire_014040627733256951610tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests:
[ERROR] org.apache.bookkeeper.bookie.storage.ldb.LocationsIndexRebuildTest
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
[ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
[ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server && /Volumes/work/devtools/jdk-11.0.16.jdk/Contents/Home/bin/java -Xmx2G -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetection.level=paranoid --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.nio.channels.spi=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.stream=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED --add-opens java.base/jdk.internal.loader=ALL-UNNAMED --add-opens java.base/sun.net.dns=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/sun.security.jca=ALL-UNNAMED --add-opens java.xml/jdk.xml.internal=ALL-UNNAMED -jar /Volumes/work/github.com/apache/bookkeeper/bookkeeper-server/target/surefire/surefirebooter14408819248685557633.jar /Volumes/w
ork/github.com/apache/bookkeeper/bookkeeper-server/target/surefire 2023-02-21T09-39-53_120-jvmRun1 surefire2887148242694623677tmp surefire_014040627733256951610tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests:
[ERROR] org.apache.bookkeeper.bookie.storage.ldb.LocationsIndexRebuildTest
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$700(ForkStarter.java:121)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:465)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:442)
[ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[ERROR] at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
(⎈|minikube:default):bookkeeper/ (branch-4.15✗) $
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org