You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/02/10 20:05:44 UTC
[GitHub] [hbase] saintstack opened a new pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
saintstack opened a new pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158
…=1C)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#discussion_r377292863
##########
File path: dev-support/jenkins-scripts/generate-hbase-website.sh
##########
@@ -120,6 +120,7 @@ if [ -z "${MAVEN_HOME}" ]; then
export PATH="${MAVEN_HOME}/bin:${PATH}"
fi
export MAVEN_OPTS="${MAVEN_OPTS} -Dmaven.repo.local=${local_repo}"
+export MAVEN_ARGS="--threads=0.5C ${MAVEN_ARGS}"
Review comment:
Was just trying to cover all the mvn references. Let me drop this for now. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-584331886
Patch adds to hbase-personality extras --threads=0.5C. Adds MAVEN_ARGS for when building site with --threads == 0.5C. Then it adds -XX:-MaxFDLimit as flag for surefire forked JVMs so we can up the file count for the jvm.... Or, oops, this seems to be a macosx special so probably won't make a difference. The build boxes are set to 16k. Lets see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-584485782
Back to 6k files. Looking at the console, the -T is not being added to mvn command runs. We're not using patched personality? I can try applying to master branch again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-584398591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 7s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 57s | branch-2 passed |
| +1 :green_heart: | compile | 3m 19s | branch-2 passed |
| +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 2m 55s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 26s | the patch passed |
| +1 :green_heart: | compile | 3m 18s | the patch passed |
| +1 :green_heart: | javac | 3m 18s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedjars | 4m 43s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 20s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 2m 54s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 106m 9s | root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 165m 14s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.master.assignment.TestCloseRegionWhileRSCrash |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1158 |
| Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit shadedjars hadoopcheck xml compile |
| uname | Linux ae4fe6f6e845 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1158/out/precommit/personality/provided.sh |
| git revision | branch-2 / 50eaa0293e |
| Default Java | 1.8.0_181 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/1/artifact/out/patch-unit-root.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/1/testReport/ |
| Max. process+thread count | 5934 (vs. ulimit of 10000) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#discussion_r377289534
##########
File path: pom.xml
##########
@@ -1428,7 +1428,7 @@
-Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=advanced
</hbase-surefire.cygwin-argLine>
<!-- Surefire argLine defaults to Linux, cygwin argLine is used in the os.windows profile -->
- <argLine>${hbase-surefire.argLine}</argLine>
+ <argLine>-XX:-MaxFDLimit ${hbase-surefire.argLine}</argLine>
Review comment:
please include a comment about why we're adding this option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack closed pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack closed pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-584333726
Removed changes to the website generating script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#discussion_r377290392
##########
File path: dev-support/jenkins-scripts/generate-hbase-website.sh
##########
@@ -120,6 +120,7 @@ if [ -z "${MAVEN_HOME}" ]; then
export PATH="${MAVEN_HOME}/bin:${PATH}"
fi
export MAVEN_OPTS="${MAVEN_OPTS} -Dmaven.repo.local=${local_repo}"
+export MAVEN_ARGS="--threads=0.5C ${MAVEN_ARGS}"
Review comment:
trying to optimize the website build at the same time as the precommit / nightly tests feels overloaded to me. How are we going to test this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-585371355
Running patches here does not insert the -T when mvn runs. Running patches out of JIRA does. Closing this then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#discussion_r377292283
##########
File path: dev-support/hbase-personality.sh
##########
@@ -140,7 +140,11 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Pass maven a -T argument. Should make it run faster. Pass conservative value.
+ # Default is one thread. 0.5C on an apache box of 24 cores and 2 executors should
+ # make for 6 threads? Lets see. See below for more on -T:
Review comment:
0.5 * 12 = 6. Let me change the title. It doesn't agree w/ what is in the patch which is trying to keep us close to current settings. Thanks @busbey
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#issuecomment-584449771
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ branch-2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 50s | branch-2 passed |
| +1 :green_heart: | compile | 3m 14s | branch-2 passed |
| +1 :green_heart: | shadedjars | 4m 40s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 2m 52s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 25s | the patch passed |
| +1 :green_heart: | compile | 3m 23s | the patch passed |
| +1 :green_heart: | javac | 3m 23s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedjars | 4m 43s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 17m 8s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 2m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 135m 58s | root in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 194m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1158 |
| Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit shadedjars hadoopcheck xml compile |
| uname | Linux cf50b659b222 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1158/out/precommit/personality/provided.sh |
| git revision | branch-2 / 50eaa0293e |
| Default Java | 1.8.0_181 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/2/testReport/ |
| Max. process+thread count | 6363 (vs. ulimit of 10000) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1158/2/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
Posted by GitBox <gi...@apache.org>.
busbey commented on a change in pull request #1158: HBASE-23823 Run maven with more than default single thread (--threads…
URL: https://github.com/apache/hbase/pull/1158#discussion_r377288653
##########
File path: dev-support/hbase-personality.sh
##########
@@ -140,7 +140,11 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Pass maven a -T argument. Should make it run faster. Pass conservative value.
+ # Default is one thread. 0.5C on an apache box of 24 cores and 2 executors should
+ # make for 6 threads? Lets see. See below for more on -T:
Review comment:
it'll be 12 I think. AFAIK each executor believes there are 24 cores on the box.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services