You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2022/07/21 00:00:51 UTC
[GitHub] [hbase] bbeaudreault opened a new pull request, #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
bbeaudreault opened a new pull request, #4638:
URL: https://github.com/apache/hbase/pull/4638
Wraps the existing codahale Histogram in a KeyValueStats object. This object tracks the global min and max sizes for the statistic, since Histogram's is subject to sampling. Additionally adds a new `-d` argument which enables printing of detailed size range counts. This can be useful for further visualizing the distribution of sizes.
Below is an example form a real HFile. Prior to this patch, the max value shown was only 25k due to the sampling done by Histogram. Additionally, the new buckets (printed when `-d` is passed) make it easy to see that there are actually a few doze quite large rows in this file.
```
Stats:
Key length:
min = 43
max = 137
mean = 104.97
stddev = 25.93
median = 105.00
75% <= 137.00
95% <= 137.00
98% <= 137.00
99% <= 137.00
99.9% <= 137.00
count = 852277
0 <= 10
785 <= 50
404643 <= 100
446849 <= 500
Val length:
min = 1
max = 2696196
mean = 1633.66
stddev = 4670.94
median = 502.00
75% <= 895.00
95% <= 7567.30
98% <= 16551.24
99% <= 24558.18
99.9% <= 52617.90
count = 852544
475 <= 1
423450 <= 500
238256 <= 1000
139640 <= 5000
19226 <= 10000
29689 <= 50000
1097 <= 100000
620 <= 500000
41 <= 750000
19 <= 1000000
31 <= 5000000
Row size (bytes):
min = 67
max = 2696274
mean = 1826.62
stddev = 5465.38
median = 619.50
75% <= 1011.75
95% <= 7299.55
98% <= 16604.48
99% <= 25200.41
99.9% <= 82035.61
count = 852277
0 <= 50
73 <= 100
274776 <= 500
359139 <= 1000
167021 <= 5000
19391 <= 10000
30066 <= 50000
1099 <= 100000
621 <= 500000
41 <= 750000
19 <= 1000000
31 <= 5000000
Row size (columns):
min = 1
max = 2
mean = 1.00
stddev = 0.03
median = 1.00
75% <= 1.00
95% <= 1.00
98% <= 1.00
99% <= 1.00
99.9% <= 1.97
count = 852277
852010 <= 1
267 <= 3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1190915790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 9s | master passed |
| +1 :green_heart: | compile | 2m 44s | master passed |
| +1 :green_heart: | checkstyle | 0m 43s | master passed |
| +1 :green_heart: | spotless | 0m 56s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 30s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 38s | the patch passed |
| +1 :green_heart: | compile | 3m 13s | the patch passed |
| +1 :green_heart: | javac | 3m 13s | the patch passed |
| -0 :warning: | checkstyle | 0m 43s | hbase-server: The patch generated 1 new + 8 unchanged - 1 fixed = 9 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 23s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 47s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. |
| | | 38m 26s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 43562c56bf1b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / da27a67a1e |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| Max. process+thread count | 66 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/1/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1191705845
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 28s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 4m 30s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 13s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 214m 10s | hbase-server in the patch passed. |
| | | 236m 2s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 401040289cfa 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/testReport/ |
| Max. process+thread count | 2489 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1208385129
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 48s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 31s | master passed |
| +1 :green_heart: | compile | 0m 34s | master passed |
| +1 :green_heart: | shadedjars | 4m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 7s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 3s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 201m 16s | hbase-server in the patch passed. |
| | | 218m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux df5bb9638448 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 54f2106f33 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/testReport/ |
| Max. process+thread count | 2976 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1208128794
@cbaenziger thanks for the input. I made changes per your request. Here's the updated format:
```
Key length:
min = 29
max = 29
mean = 29.00
stddev = 0.00
median = 29.00
75% <= 29.00
95% <= 29.00
98% <= 29.00
99% <= 29.00
99.9% <= 29.00
count = 1000
(range <= count):
10 <= 0
50 <= 1000
Val length:
min = 3
max = 3
mean = 3.00
stddev = 0.00
median = 3.00
75% <= 3.00
95% <= 3.00
98% <= 3.00
99% <= 3.00
99.9% <= 3.00
count = 1000
(range <= count):
1 <= 0
3 <= 1000
Row size (bytes):
min = 40
max = 40
mean = 40.00
stddev = 0.00
median = 40.00
75% <= 40.00
95% <= 40.00
98% <= 40.00
99% <= 40.00
99.9% <= 40.00
count = 1000
(range <= count):
10 <= 0
50 <= 1000
Row size (columns):
min = 1
max = 1
mean = 1.00
stddev = 0.00
median = 1.00
75% <= 1.00
95% <= 1.00
98% <= 1.00
99% <= 1.00
99.9% <= 1.00
count = 1000
(range <= count):
1 <= 1000
Key of biggest row: row_00000000
```
Worth noting that the new `range <= count` sections only show up if you enable the new `-d` arg.
To your last question:
> Lastly, this may be a question/solution chasing a problem, but will this show a bi-modal set of ranges clearly (e.g. if I have keys of 50 bytes and keys of 5,000 bytes only) or will the elided bounding ranges be needed to point that out? Or is that what [line 845](https://github.com/apache/hbase/pull/4638/commits/820104f375217e2cad2cfa0d3f3c4631a6ec6599#diff-35423d30ce8e7b499844db2cd71563dea12517322eb476239bc7867ccd96ac6dL840-L841) is doing already?
Yes, line 845 attempts to provide some context while not printing all of the 0 ranges between 2 values. Probably the most clear would be to print all ranges, but that'd also be way unnecessarily verbose in most cases. So I tried to only print the context necessary, i think it should be pretty intuitive once someone sees the output for a few files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1199259785
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 16s | master passed |
| +1 :green_heart: | compile | 2m 12s | master passed |
| +1 :green_heart: | checkstyle | 0m 30s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 17s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 10s | the patch passed |
| +1 :green_heart: | compile | 2m 13s | the patch passed |
| +1 :green_heart: | javac | 2m 13s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 30s | hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. |
| | | 31m 28s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 7bb663f5d951 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 811f0e7d2a |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault merged pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
bbeaudreault merged PR #4638:
URL: https://github.com/apache/hbase/pull/4638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4638:
URL: https://github.com/apache/hbase/pull/4638#discussion_r930431234
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -760,6 +830,37 @@ private void printHistogram(Histogram histogram) {
output.printf(locale, " 99%% <= %2.2f%n", snapshot.get99thPercentile());
output.printf(locale, " 99.9%% <= %2.2f%n", snapshot.get999thPercentile());
output.printf(locale, " count = %d%n", histogram.getCount());
+
Review Comment:
@cbaenziger just pushed a comment, let me know if it clears anything up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1191468817
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 26s | master passed |
| +1 :green_heart: | compile | 2m 15s | master passed |
| +1 :green_heart: | checkstyle | 0m 32s | master passed |
| +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 20s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 10s | the patch passed |
| +1 :green_heart: | compile | 2m 13s | the patch passed |
| +1 :green_heart: | javac | 2m 13s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 44s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 32m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 76f8bb5e09c1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 60 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196165079
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 26s | master passed |
| +1 :green_heart: | compile | 0m 35s | master passed |
| +1 :green_heart: | shadedjars | 4m 0s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 14s | the patch passed |
| +1 :green_heart: | compile | 0m 35s | the patch passed |
| +1 :green_heart: | javac | 0m 35s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 3s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 206m 34s | hbase-server in the patch failed. |
| | | 223m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux bc7b6a1a5f8d 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9ba868d2ad |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/testReport/ |
| Max. process+thread count | 2400 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4638:
URL: https://github.com/apache/hbase/pull/4638#discussion_r929942907
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -640,49 +722,39 @@ public String toString() {
if (prevCell == null) return "no data available for statistics";
// Dump the metrics to the output stream
- simpleReporter.stop();
Review Comment:
Yea, so this previously had SimpleReporter extend ScheduledReporter. In my opinion this was overkill. The benefits of ScheduledReporter are:
- it has a thread which can print the statistics periodically over time. You need to call reporter.start() for that.
- It can do various filters on the metrics, which is mostly useful when you have larger shared MetricsRegistry and maybe only care about certain metrics.
- It can simplify the collection of multiple different metric types (counters, gauges, etc). Including some extra pre-processing that can be applied.
We weren't using any of these features, so I think it was unnecessary to use ScheduledReporter. I would have been fine to keep it in place just to simplify this PR, but there were issues with ScheduledReporter.
Our primary goal here was to add a global min/max for each of the metrics collected. Since ScheduledReporter is based around processing codahale metrics, it's hard to supplement those metrics with additional metadata. I think the "codahale-native" way for me to do this would have been to add a new Gauge for all of the existing Histograms. Then, in the `report` method below we'd get those gauges alongside the histograms. The annoying thing with that is we'd have to manually match up the names somehow, and the names have to be unique. So maybe I'd name the histogram `Key length` and then the gauge `Key length - max` and have to do some string munging to link the 2 together.
That felt pretty hacky and all for the purposes of keeping a thing we don't need/use anyway. I also tried some other hacks to keep it, liking having a static map which kept them together, but this felt similarly hacky. So that's why I ripped out the ScheduledReporter.
With ScheduledReporter, the `report` method is triggered when you call `stop` (as well as periodically by the reporting thread if you had called `start()`, which we don't). So you're correct, that's why we used to have to call `stop()`. Now that we don't rely on ScheduledReporter, we don't need to call that at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196037296
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 9s | master passed |
| +1 :green_heart: | compile | 2m 47s | master passed |
| +1 :green_heart: | checkstyle | 0m 41s | master passed |
| +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 38s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed |
| +1 :green_heart: | compile | 2m 40s | the patch passed |
| +1 :green_heart: | javac | 2m 40s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 32s | hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 48s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 44s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. |
| | | 38m 54s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 8c54007870f6 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9ba868d2ad |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cbaenziger commented on a diff in pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
cbaenziger commented on code in PR #4638:
URL: https://github.com/apache/hbase/pull/4638#discussion_r935013421
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -760,6 +830,37 @@ private void printHistogram(Histogram histogram) {
output.printf(locale, " 99%% <= %2.2f%n", snapshot.get99thPercentile());
output.printf(locale, " 99.9%% <= %2.2f%n", snapshot.get999thPercentile());
output.printf(locale, " count = %d%n", histogram.getCount());
+
Review Comment:
Thanks! The code and review comments makes it more clear what is intended. I think the output will need some explanation of what is being displayed after the count.
Right now, it looks like it's on lines [864-866](https://github.com/apache/hbase/pull/4638/commits/820104f375217e2cad2cfa0d3f3c4631a6ec6599#diff-35423d30ce8e7b499844db2cd71563dea12517322eb476239bc7867ccd96ac6dL860-L862) printing:
`<count> <= <range>`
While all the other lines print:
`<descriptor> <= <value from data>`
Could the range count be reversed to match the layout of the other lines?
`<range> <= <count>`
Lastly, this may be a question/solution chasing a problem, but will this show a bi-modal set of ranges clearly (e.g. if I have keys of 50 bytes and keys of 5,000 bytes only) or will the elided bounding ranges be needed to point that out? Or is that what [line 845](https://github.com/apache/hbase/pull/4638/commits/820104f375217e2cad2cfa0d3f3c4631a6ec6599#diff-35423d30ce8e7b499844db2cd71563dea12517322eb476239bc7867ccd96ac6dL840-L841) is doing already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a diff in pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
bbeaudreault commented on code in PR #4638:
URL: https://github.com/apache/hbase/pull/4638#discussion_r929944933
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -760,6 +830,37 @@ private void printHistogram(Histogram histogram) {
output.printf(locale, " 99%% <= %2.2f%n", snapshot.get99thPercentile());
output.printf(locale, " 99.9%% <= %2.2f%n", snapshot.get999thPercentile());
output.printf(locale, " count = %d%n", histogram.getCount());
+
Review Comment:
The min/max change is handled above on lines 822/823. The below logic prints the counts-by-range, i.e. the last 4 lines here:
```
Key length:
min = 43
max = 137
mean = 104.97
stddev = 25.93
median = 105.00
75% <= 137.00
95% <= 137.00
98% <= 137.00
99% <= 137.00
99.9% <= 137.00
count = 852277
0 <= 10
785 <= 50
404643 <= 100
446849 <= 500
```
I could add a comment here just to describe what's happening. It's similar to how we handle range counts in JMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1191721009
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 44s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 50s | master passed |
| +1 :green_heart: | compile | 1m 8s | master passed |
| +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 59s | the patch passed |
| +1 :green_heart: | compile | 0m 53s | the patch passed |
| +1 :green_heart: | javac | 0m 53s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 8s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 226m 41s | hbase-server in the patch passed. |
| | | 252m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a61815954c50 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 31fc97edee |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/testReport/ |
| Max. process+thread count | 2753 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/2/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196380083
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 27s | master passed |
| +1 :green_heart: | compile | 0m 36s | master passed |
| +1 :green_heart: | shadedjars | 4m 2s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 12s | the patch passed |
| +1 :green_heart: | compile | 0m 35s | the patch passed |
| +1 :green_heart: | javac | 0m 35s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 2s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 202m 27s | hbase-server in the patch failed. |
| | | 219m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 9745001003a3 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80b366817e |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/testReport/ |
| Max. process+thread count | 2617 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196398149
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 56s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 24s | master passed |
| +1 :green_heart: | compile | 1m 9s | master passed |
| +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 21s | the patch passed |
| +1 :green_heart: | compile | 1m 7s | the patch passed |
| +1 :green_heart: | javac | 1m 7s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 15s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 213m 7s | hbase-server in the patch failed. |
| | | 238m 16s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux b9d665c321aa 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80b366817e |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/testReport/ |
| Max. process+thread count | 2808 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1199643484
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 12s | master passed |
| +1 :green_heart: | compile | 0m 35s | master passed |
| +1 :green_heart: | shadedjars | 4m 7s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 17s | the patch passed |
| +1 :green_heart: | compile | 0m 36s | the patch passed |
| +1 :green_heart: | javac | 0m 36s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 5s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 204m 24s | hbase-server in the patch passed. |
| | | 221m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a4fabe2db5af 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 811f0e7d2a |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/testReport/ |
| Max. process+thread count | 2551 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1199641836
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 34s | master passed |
| +1 :green_heart: | compile | 0m 46s | master passed |
| +1 :green_heart: | shadedjars | 3m 43s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 47s | the patch passed |
| +1 :green_heart: | javac | 0m 47s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 42s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 202m 40s | hbase-server in the patch passed. |
| | | 220m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 7423efb4d9c0 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 811f0e7d2a |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/testReport/ |
| Max. process+thread count | 2644 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/5/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1208172393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 20s | master passed |
| +1 :green_heart: | compile | 2m 18s | master passed |
| +1 :green_heart: | checkstyle | 0m 31s | master passed |
| +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 17s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 15s | the patch passed |
| +1 :green_heart: | compile | 2m 10s | the patch passed |
| +1 :green_heart: | javac | 2m 10s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 30s | hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 11m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 22s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. |
| | | 32m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 2f5574db23eb 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 54f2106f33 |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1208387980
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 7s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 44s | master passed |
| +1 :green_heart: | compile | 0m 46s | master passed |
| +1 :green_heart: | shadedjars | 3m 45s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 46s | the patch passed |
| +1 :green_heart: | javac | 0m 46s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 44s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 203m 17s | hbase-server in the patch passed. |
| | | 221m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d906e0d032c9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 54f2106f33 |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/testReport/ |
| Max. process+thread count | 2840 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/6/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] cbaenziger commented on a diff in pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
cbaenziger commented on code in PR #4638:
URL: https://github.com/apache/hbase/pull/4638#discussion_r929675636
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -760,6 +830,37 @@ private void printHistogram(Histogram histogram) {
output.printf(locale, " 99%% <= %2.2f%n", snapshot.get99thPercentile());
output.printf(locale, " 99.9%% <= %2.2f%n", snapshot.get999thPercentile());
output.printf(locale, " count = %d%n", histogram.getCount());
+
Review Comment:
The below block is really unobvious to me how it is (presumably) printing mins and maxes but it may just be late for me. I'll take another look tomorrow. Perhaps a comment of what the process being implemented is could be of assistance?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:
##########
@@ -640,49 +722,39 @@ public String toString() {
if (prevCell == null) return "no data available for statistics";
// Dump the metrics to the output stream
- simpleReporter.stop();
Review Comment:
This was unexpected at first to me as I am not very CodaHale aware but I questioned why remove the `simpleReporter.stop()`? (It does seem odd to stop the reporter in a `toString()` implementation but I think this was a hack to clean the reporter up when emitting its metrics?)
As I understand now `simpleReporter` is no longer based on a CodaHale metric class with any thread or other state to clean-up though so no need for a `stop()` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196171226
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 58s | master passed |
| +1 :green_heart: | compile | 0m 47s | master passed |
| +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 48s | the patch passed |
| +1 :green_heart: | javac | 0m 48s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 48s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 218m 9s | hbase-server in the patch failed. |
| | | 236m 47s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 22268db5610a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9ba868d2ad |
| Default Java | AdoptOpenJDK-11.0.10+9 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/testReport/ |
| Max. process+thread count | 2464 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/3/console |
| versions | git=2.17.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4638: HBASE-27224 HFile tool statistic sampling produces misleading results
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4638:
URL: https://github.com/apache/hbase/pull/4638#issuecomment-1196263168
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 53s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 1s | master passed |
| +1 :green_heart: | compile | 2m 59s | master passed |
| +1 :green_heart: | checkstyle | 0m 43s | master passed |
| +1 :green_heart: | spotless | 0m 59s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 58s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | the patch passed |
| +1 :green_heart: | compile | 2m 39s | the patch passed |
| +1 :green_heart: | javac | 2m 39s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 33s | hbase-server: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. |
| +1 :green_heart: | spotless | 0m 50s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. |
| | | 41m 1s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4638 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux f00692c15f69 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 80b366817e |
| Default Java | AdoptOpenJDK-1.8.0_282-b08 |
| Max. process+thread count | 64 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4638/4/console |
| versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org