You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2022/07/20 23:39:30 UTC
[GitHub] [hbase] bbeaudreault opened a new pull request, #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
bbeaudreault opened a new pull request, #4637:
URL: https://github.com/apache/hbase/pull/4637
Example output (with log4j log formatting prefixes removed for clarity):
```
Bucket allocator statistics follow:
Free bytes=5325249536; used bytes=35391673344; total bytes=40716922880; waisted bytes=355258368; completelyFreeBuckets=843
Object size 33792; used=195000; free=114; total=195114; waisted bytes=10741760; full buckets=1048
Object size 66560; used=275656; free=46; total=275702; waisted bytes=114128896; full buckets=2932
Object size 99328; used=76785; free=12; total=76797; waisted bytes=46185472; full buckets=1218
Object size 132096; used=6090; free=20; total=6110; waisted bytes=11315200; full buckets=129
Object size 525312; used=3303; free=8; total=3311; waisted bytes=155653120; full buckets=300
Object size 787456; used=152; free=2; total=154; waisted bytes=17233920; full buckets=21
Object size 1573888; used=107; free=3373; total=3480; waisted bytes=0; full buckets=26
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1190896415
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 27s | master passed |
| +1 :green_heart: | compile | 2m 17s | master passed |
| +1 :green_heart: | checkstyle | 0m 34s | master passed |
| +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 21s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 13s | the patch passed |
| +1 :green_heart: | compile | 2m 14s | the patch passed |
| +1 :green_heart: | javac | 2m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 32m 26s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux e7b355b611a7 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / da27a67a1e |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927598647
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
Happy to, and feel free to tell me if I'm misunderstanding here. I'm also learning the BucketCache recently.
- `bucketCapacity` is calculated once for the whole cache. It's defined as `4 * largestBucketSize`.
- Then, each bucket is allocated to one of a configured number of bucket sizes
- The configured bucket sizes may not divide into the global `bucketCapacity` well, leaving a remainder.
- That size of that remainder will vary for each bucket size, and any bucket allocated to that size will have that much waisted space.
- Typically each bucket size will have 1 `freeBucket` (which is probably better defined as a `fillingBucket`) and a number of `fullBuckets`. A fullBucket isn't actually full, depending on the block size it's allocated to. The remainder for the block size will be empty/unused.
- That remainder should be the same for all buckets of a particular size, because the remainder is based on the configured bucket size.
Let's use my example snippet from above:
```
Bucket allocator statistics follow:
Free bytes=5325249536; used bytes=35391673344; total bytes=40716922880; waisted bytes=355258368; completelyFreeBuckets=843
Object size 33792; used=195000; free=114; total=195114; waisted bytes=10741760; full buckets=1048
Object size 66560; used=275656; free=46; total=275702; waisted bytes=114128896; full buckets=2932
Object size 99328; used=76785; free=12; total=76797; waisted bytes=46185472; full buckets=1218
Object size 132096; used=6090; free=20; total=6110; waisted bytes=11315200; full buckets=129
Object size 525312; used=3303; free=8; total=3311; waisted bytes=155653120; full buckets=300
Object size 787456; used=152; free=2; total=154; waisted bytes=17233920; full buckets=21
Object size 1573888; used=107; free=3373; total=3480; waisted bytes=0; full buckets=26
```
- I've noticed across a bunch of hosts that the 525312 bucket has the most waist based on this configuration.
- The largest bucket size is 157388, so the bucketCapacity is 6295552
- So using 525312 as an example, it will always allocate blocks in those buckets with a size of 525312, even if the actual block size isn't exactly that amount [*].
- `6295552 % 525312 = 517120`. So given that we always allocate blocks with size 525312, we basically are missing out on ~1 block per bucket allocated for this size. There are only approx `6295552 / 525312 = 11` blocks fitting into this bucket size, so that almost 10% waist.
- Checking 66560, which is next most waistful but has many more buckets. `6295552 % 66560 = 38912`.
- So each bucket will waist about half a block. For that block size, it can fit 94 blocks per bucket. So waisting half a block is much more efficient than above.
Buckets can be reallocated over time. Once they become a `completelyFreeBucket`, the next allocation that needs one, of any size, will take that bucket and reconfigure it for its purpose. At that point the amount of waist for that bucket would change based on the new block size.
-----
[*] getting back to "even if the actual block size isn't exactly that amount". This speaks to another source of waist which is harder to calculate. So the waistBytes is actually very much an underestimation, but I'm not sure by how much. Let's say you have a block with 150k size. That doesn't fit into the 129k bucket, so it has to go into the 513k bucket. But even though the block size is 150k, we need to allocate a full 513k, leaving 363k of waisted space.
I say this is harder to calculate, but of course it'd be relatively easy. When we call `roundUpToBucketSizeInfo` we could just subtract the blockSize from the bucketSize, and then add that diff to a histogram. But I was just thinking this might be a very hot codepath and adding a histogram there will be a lot more expensive than what I currently have since currently the statistics are calculated totally off the hot path.
I imagine maybe the unified.encoded.blocksize could help with this problem, but also part of the reason for so much upward skew for us is because we have some users writing large rows. So may not entirely help us at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191906116
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 20s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 27s | master passed |
| +1 :green_heart: | compile | 0m 41s | master passed |
| +1 :green_heart: | shadedjars | 3m 44s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 13s | the patch passed |
| +1 :green_heart: | compile | 0m 40s | the patch passed |
| +1 :green_heart: | javac | 0m 40s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 44s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 206m 51s | hbase-server in the patch passed. |
| | | 223m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3eb55edcdfe6 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/testReport/ |
| Max. process+thread count | 2453 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193478368
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 28s | master passed |
| +1 :green_heart: | compile | 2m 11s | master passed |
| +1 :green_heart: | checkstyle | 0m 29s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 14s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 7s | the patch passed |
| +1 :green_heart: | compile | 2m 14s | the patch passed |
| +1 :green_heart: | javac | 2m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 28s | hbase-server: The patch generated 0 new + 27 unchanged - 2 fixed = 27 total (was 29) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 47s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. |
| | | 31m 58s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 16ed0b5e010c 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0ae42ddaee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 60 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1192852256
Thanks @wchevreuil, can you give it one more look?
I realized LongAdder is not necessary because both increment/decrement are behind synchronization. Changed it to a normal long.
I also added a bunch of javadoc to help people understand what they're looking at.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193067921
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 44s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 31s | master passed |
| +1 :green_heart: | compile | 0m 35s | master passed |
| +1 :green_heart: | shadedjars | 4m 5s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | the patch passed |
| +1 :green_heart: | compile | 0m 51s | the patch passed |
| +1 :green_heart: | javac | 0m 51s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 49s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 206m 39s | hbase-server in the patch passed. |
| | | 226m 26s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a0dd5881bbf7 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/testReport/ |
| Max. process+thread count | 2771 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191716124
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 50s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 51s | master passed |
| +1 :green_heart: | compile | 0m 40s | master passed |
| +1 :green_heart: | shadedjars | 4m 25s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 22s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 233m 50s | hbase-server in the patch failed. |
| | | 253m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 57fd4c8fcd45 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/testReport/ |
| Max. process+thread count | 2729 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927756440
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
> So using 525312 as an example, it will always allocate blocks in those buckets with a size of 525312, even if the actual block size isn't exactly that amount [*].
> 6295552 % 525312 = 517120. So given that we always allocate blocks with size 525312, we basically are missing out on ~1 block per bucket allocated for this size. There are only approx 6295552 / 525312 = 11 blocks fitting into this bucket size, so that almost 10% waist.
Ok, so this _waste_ here is a best case scenario, not taking into account the real size of allocated blocks on each bucket? Considering that real world cases may not have perfectly sized 512KB blocks, fragmentation would be even higher?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927797150
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
I just implemented the above in my latest patch. I'm thinking a LongAdder increment/decrement is probably not going to make a noticeable difference next to everything else.
Here's an example output from the test, unfortunately I don't have a real-world example:
```
Free bytes=20314112; used bytes=10435584; total bytes=30749696; wasted bytes=70656; fragmentation bytes=728064; completelyFreeBuckets=10
Object size 5120; used=410; free=0; total=410; wasted bytes=2048; fragmentation bytes=419840, full buckets=1
Object size 9216; used=228; free=0; total=228; wasted bytes=0; fragmentation bytes=233472, full buckets=1
Object size 17408; used=0; free=120; total=120; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 33792; used=0; free=62; total=62; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 41984; used=0; free=50; total=50; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 50176; used=0; free=41; total=41; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 58368; used=0; free=36; total=36; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 66560; used=31; free=0; total=31; wasted bytes=37888; fragmentation bytes=31744, full buckets=1
Object size 99328; used=42; free=0; total=42; wasted bytes=30720; fragmentation bytes=43008, full buckets=2
Object size 132096; used=0; free=15; total=15; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 197632; used=0; free=10; total=10; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 263168; used=0; free=7; total=7; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 394240; used=0; free=5; total=5; wasted bytes=0; fragmentation bytes=0, full buckets=0
Object size 525312; used=0; free=4; total=4; wasted bytes=0; fragmentation bytes=0, full buckets=0
```
We could combine wasted + fragmented. Probably confusing to have both, but also useful once you understand the difference (which we can document)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927764123
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
yea, correct. fragmentation could make this worse, and its harder to cheaply calculate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927757722
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
nit: I guess you mean `wastedBytes` not `waistedBytes` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193044079
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 17s | master passed |
| +1 :green_heart: | compile | 2m 12s | master passed |
| +1 :green_heart: | checkstyle | 0m 31s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 17s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 11s | the patch passed |
| +1 :green_heart: | compile | 2m 10s | the patch passed |
| +1 :green_heart: | javac | 2m 10s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | hbase-server: The patch generated 0 new + 27 unchanged - 2 fixed = 27 total (was 29) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. |
| | | 31m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux b42939226ffb 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927766995
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
if you have ideas, i'd be happy to try adding that. One thought I had was to add a LongAdder to each BucketSizeInfo, and increment it in `allocateBlock(int blockSize)`. But really we should also be decrementing somewhere. Possibly in `freeBlock`, but we'd need to add the blockSize as an argument there. And there's a question of the impact on performance (probably small/worth it?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191462579
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 29s | master passed |
| +1 :green_heart: | compile | 2m 11s | master passed |
| +1 :green_heart: | checkstyle | 0m 28s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 15s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 5s | the patch passed |
| +1 :green_heart: | compile | 2m 12s | the patch passed |
| +1 :green_heart: | javac | 2m 12s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 52s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 17s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. |
| | | 32m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 7bdace171054 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927759151
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -509,46 +522,75 @@ public long itemSize() {
return itemSize;
}
- public IndexStatistics(long free, long used, long itemSize) {
- setTo(free, used, itemSize);
+ public int fullBuckets() {
+ return fullBuckets;
+ }
+
+ public int completelyFreeBuckets() {
+ return completelyFreeBuckets;
+ }
+
+ public long waistedBytes() {
+ return waistedBytes;
+ }
+
+ public IndexStatistics(long free, long used, long itemSize, int fullBuckets,
+ int completelyFreeBuckets, long waistedBytes) {
+ setTo(free, used, itemSize, fullBuckets, completelyFreeBuckets, waistedBytes);
}
public IndexStatistics() {
- setTo(-1, -1, 0);
+ setTo(-1, -1, 0, 0, 0, 0);
}
- public void setTo(long free, long used, long itemSize) {
+ public void setTo(long free, long used, long itemSize, int fullBuckets,
+ int completelyFreeBuckets, long waistedBytes) {
this.itemSize = itemSize;
this.freeCount = free;
this.usedCount = used;
this.totalCount = free + used;
+ this.fullBuckets = fullBuckets;
+ this.completelyFreeBuckets = completelyFreeBuckets;
+ this.waistedBytes = waistedBytes;
}
}
public Bucket[] getBuckets() {
return this.buckets;
}
- void logStatistics() {
+ void logDebugStatistics() {
+ if (!LOG.isDebugEnabled()) {
+ return;
+ }
+
IndexStatistics total = new IndexStatistics();
IndexStatistics[] stats = getIndexStatistics(total);
- LOG.info("Bucket allocator statistics follow:\n");
- LOG.info(" Free bytes=" + total.freeBytes() + "+; used bytes=" + total.usedBytes()
- + "; total bytes=" + total.totalBytes());
+ LOG.debug("Bucket allocator statistics follow:");
+ LOG.debug(
+ " Free bytes={}; used bytes={}; total bytes={}; waisted bytes={}; completelyFreeBuckets={}",
Review Comment:
nit: `s/waisted/wasted/g`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193594881
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 12s | master passed |
| +1 :green_heart: | compile | 0m 35s | master passed |
| +1 :green_heart: | shadedjars | 4m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 10s | the patch passed |
| +1 :green_heart: | compile | 0m 35s | the patch passed |
| +1 :green_heart: | javac | 0m 35s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 206m 5s | hbase-server in the patch passed. |
| | | 222m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux faed9f8c8b0c 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0ae42ddaee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/testReport/ |
| Max. process+thread count | 2624 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191747190
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 32s | master passed |
| +1 :green_heart: | compile | 2m 14s | master passed |
| +1 :green_heart: | checkstyle | 0m 32s | master passed |
| +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 20s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 13s | the patch passed |
| +1 :green_heart: | compile | 2m 14s | the patch passed |
| +1 :green_heart: | javac | 2m 14s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 40s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 32m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux f6699193fbb3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193669366
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 59s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 53s | master passed |
| +1 :green_heart: | compile | 1m 4s | master passed |
| +1 :green_heart: | shadedjars | 4m 43s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 40s | the patch passed |
| +1 :green_heart: | compile | 0m 59s | the patch passed |
| +1 :green_heart: | javac | 0m 59s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 32s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 307m 27s | hbase-server in the patch failed. |
| | | 332m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 64db868ad1a6 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 0ae42ddaee |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/testReport/ |
| Max. process+thread count | 2573 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1193065907
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 28s | master passed |
| +1 :green_heart: | compile | 0m 42s | master passed |
| +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 40s | the patch passed |
| +1 :green_heart: | javac | 0m 40s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 46s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 191m 36s | hbase-server in the patch failed. |
| | | 208m 50s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 48b4e5b62f53 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/testReport/ |
| Max. process+thread count | 2799 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191681816
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 55s | master passed |
| +1 :green_heart: | compile | 0m 43s | master passed |
| +1 :green_heart: | shadedjars | 4m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 36s | the patch passed |
| +1 :green_heart: | compile | 0m 42s | the patch passed |
| +1 :green_heart: | javac | 0m 42s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 55s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 199m 8s | hbase-server in the patch passed. |
| | | 218m 13s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux ce259e247141 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/testReport/ |
| Max. process+thread count | 2602 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1191909941
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 21s | master passed |
| +1 :green_heart: | compile | 0m 48s | master passed |
| +1 :green_heart: | shadedjars | 4m 8s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 7s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 33s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 205m 7s | hbase-server in the patch failed. |
| | | 227m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a4b1ac6872a2 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/testReport/ |
| Max. process+thread count | 2826 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a diff in pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on code in PR #4637:
URL: https://github.com/apache/hbase/pull/4637#discussion_r927557082
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java:
##########
@@ -247,12 +247,24 @@ public void freeBlock(Bucket b, long offset) {
public synchronized IndexStatistics statistics() {
long free = 0, used = 0;
+ int full = 0;
for (Object obj : bucketList.keySet()) {
Bucket b = (Bucket) obj;
free += b.freeCount();
used += b.usedCount();
+ if (!b.hasFreeSpace()) {
+ full++;
+ }
}
- return new IndexStatistics(free, used, bucketSizes[sizeIndex]);
+ int bucketObjectSize = bucketSizes[sizeIndex];
+ // this is most likely to always be 1 or 0
+ int fillingBuckets = Math.max(0, freeBuckets.size() - completelyFreeBuckets.size());
+ // if bucket capacity is not perfectly divisible by a bucket's object size, there will
+ // be some left over per bucket. for some object sizes this may be large enough to be
+ // non-trivial and worth tuning by choosing a more divisible object size.
+ long waistedBytes = (bucketCapacity % bucketObjectSize) * (full + fillingBuckets);
Review Comment:
Mind teach me briefly on this math here? Aren't `bucketCapacity` and `bucketObjectSize` computed for each individual bucket? If so, why are we multiplying by the number of buckets that already have some data? Wouldn't this (bucketCapacity % bucketObjectSize) differ for each bucket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1192759220
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 11s | master passed |
| +1 :green_heart: | compile | 2m 18s | master passed |
| +1 :green_heart: | checkstyle | 0m 31s | master passed |
| +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 17s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 13s | the patch passed |
| +1 :green_heart: | compile | 2m 11s | the patch passed |
| +1 :green_heart: | javac | 2m 11s | the patch passed |
| -0 :warning: | checkstyle | 0m 31s | hbase-server: The patch generated 2 new + 27 unchanged - 2 fixed = 29 total (was 29) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. |
| | | 31m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux a433c75dabc5 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 60 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4637:
URL: https://github.com/apache/hbase/pull/4637#issuecomment-1192892448
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 21s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 14s | master passed |
| +1 :green_heart: | compile | 0m 39s | master passed |
| +1 :green_heart: | shadedjars | 3m 46s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 17s | the patch passed |
| +1 :green_heart: | compile | 0m 39s | the patch passed |
| +1 :green_heart: | javac | 0m 39s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 44s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 206m 23s | hbase-server in the patch passed. |
| | | 222m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4637 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9eb98e8a17fa 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/testReport/ |
| Max. process+thread count | 2494 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4637/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault merged pull request #4637: HBASE-27225 Add BucketAllocator bucket size statistic logging
Posted by GitBox <gi...@apache.org>.
bbeaudreault merged PR #4637:
URL: https://github.com/apache/hbase/pull/4637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org