You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2022/11/21 17:32:54 UTC
[GitHub] [hbase] 2005hithlj opened a new pull request, #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
2005hithlj opened a new pull request, #4891:
URL: https://github.com/apache/hbase/pull/4891
https://issues.apache.org/jira/browse/HBASE-27483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1331608097
@taklwu sir, I'm sorry to have kept you waiting so long.
I have a new commit, could you take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1549072134
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 43s | master passed |
| +1 :green_heart: | compile | 0m 54s | master passed |
| +1 :green_heart: | shadedjars | 4m 46s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | hbase-hadoop-compat in the patch passed. |
| -1 :x: | unit | 211m 45s | hbase-server in the patch failed. |
| | | 236m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 242752cd3bd1 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 93ddd7060d |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/testReport/ |
| Max. process+thread count | 4675 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1055425790
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -313,9 +341,24 @@ public void run() {
tempVal += store.getMemstoreOnlyRowReadsCount();
}
readsOnlyFromMemstore.put(store.getColumnFamilyName(), tempVal);
+
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
Review Comment:
> Use `long` instead of `Long` as later we will do calculation?
OK, sir.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -313,9 +341,24 @@ public void run() {
tempVal += store.getMemstoreOnlyRowReadsCount();
}
readsOnlyFromMemstore.put(store.getColumnFamilyName(), tempVal);
+
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
Review Comment:
OK, sir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1322646388
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 37s | master passed |
| +1 :green_heart: | compile | 0m 51s | master passed |
| +1 :green_heart: | shadedjars | 3m 54s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 52s | the patch passed |
| +1 :green_heart: | javac | 0m 52s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 51s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 194m 33s | hbase-server in the patch passed. |
| | | 215m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 0856499cfef2 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9462933179 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/testReport/ |
| Max. process+thread count | 2685 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
taklwu commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1028805044
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -59,15 +61,28 @@ public class MetricsRegionWrapperImpl implements MetricsRegionWrapper, Closeable
private long maxCompactionQueueSize;
private Map<String, Long> readsOnlyFromMemstore;
private Map<String, Long> mixedReadsOnStore;
+ private Map<Integer, Long> storeFilesAccessedDaysAndSize;
+ private int[] storeFilesAccessedDaysThresholds;
private ScheduledFuture<?> regionMetricsUpdateTask;
+ public static long ONE_DAY_MS = 24 * 3600 * 1000;
+ public static final String STOREFILES_ACCESSED_DAYS_THRESHOLDS =
+ "hbase.region.storefiles.accessed.days.thresholds";
+ public static final int[] STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT = { 7, 30, 90 };
Review Comment:
One more question, if we are using cache / bucket cache heavily, will this metric retain at the time the store file flushes or opened for cache? It’s fine as is without caring the cache , just want to learn more here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1331630738
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 24s | master passed |
| +1 :green_heart: | compile | 2m 37s | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | master passed |
| +1 :green_heart: | spotless | 0m 37s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 35s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | the patch passed |
| +1 :green_heart: | compile | 2m 38s | the patch passed |
| +1 :green_heart: | javac | 2m 38s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 8m 47s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 34m 49s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux f363331a39c0 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3ffc0b5552 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 84 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1194536872
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
+ for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+ Long accessTime = pair.getFirst();
+ Long size = pair.getSecond();
+ for (int threshold : storeFilesAccessedDaysThresholds) {
+ Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+ if ((now - accessTime) >= threshold * ONE_DAY_MS) {
Review Comment:
OK, so the metric here is 'the number if store files which are not accessed by at least xx days'? Then I think we;d better have a better name for it? Looking at 'storeFilesAccessedDays', I can not get the meaning unless you explain it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
taklwu commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1028805044
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -59,15 +61,28 @@ public class MetricsRegionWrapperImpl implements MetricsRegionWrapper, Closeable
private long maxCompactionQueueSize;
private Map<String, Long> readsOnlyFromMemstore;
private Map<String, Long> mixedReadsOnStore;
+ private Map<Integer, Long> storeFilesAccessedDaysAndSize;
+ private int[] storeFilesAccessedDaysThresholds;
private ScheduledFuture<?> regionMetricsUpdateTask;
+ public static long ONE_DAY_MS = 24 * 3600 * 1000;
+ public static final String STOREFILES_ACCESSED_DAYS_THRESHOLDS =
+ "hbase.region.storefiles.accessed.days.thresholds";
+ public static final int[] STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT = { 7, 30, 90 };
Review Comment:
One more question, if we are using cache / bucket cache heavily, will this metric retain at the time the store file flushes or opened for cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1544053512
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 52s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 57s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 4m 19s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 45s | the patch passed |
| +1 :green_heart: | compile | 0m 52s | the patch passed |
| +1 :green_heart: | javac | 0m 52s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 18s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 227m 56s | hbase-server in the patch passed. |
| | | 252m 4s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 8e72edfcbb04 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5cea8112fd |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/testReport/ |
| Max. process+thread count | 2508 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache9 commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1056735714
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
return getStoreFileAgeStream().average();
}
+ @Override
+ public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+ Collection<HStoreFile> storeFiles = this.storeEngine.getStoreFileManager().getStorefiles();
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+ for (HStoreFile sf : storeFiles) {
+ if (sf.getReader() == null) {
+ continue;
+ }
+ FileStatus fileStatus;
+ try {
+ fileStatus = sf.getFileInfo().getFileStatus();
Review Comment:
Will this introduce a request spike to NN? Usually after opening a store file we will not goto NN any more...
I wonder if we can maintain this information inside HBase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1364411484
@Apache9 sir. Could you take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1322461997
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 17s | master passed |
| +1 :green_heart: | compile | 3m 0s | master passed |
| +1 :green_heart: | checkstyle | 0m 43s | master passed |
| +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 59s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 53s | the patch passed |
| +1 :green_heart: | compile | 2m 45s | the patch passed |
| +1 :green_heart: | javac | 2m 45s | the patch passed |
| -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 9m 39s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| -1 :x: | spotbugs | 1m 36s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 35m 59s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hbase-server |
| | org.apache.hadoop.hbase.regionserver.MetricsRegionWrapperImpl.ONE_DAY_MS isn't final but should be At MetricsRegionWrapperImpl.java:be At MetricsRegionWrapperImpl.java:[line 69] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 02b4c5334e98 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9462933179 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt |
| spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html |
| Max. process+thread count | 82 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1543751479
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 33s | master passed |
| +1 :green_heart: | compile | 2m 38s | master passed |
| +1 :green_heart: | checkstyle | 0m 39s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 41s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 17s | the patch passed |
| +1 :green_heart: | compile | 2m 35s | the patch passed |
| +1 :green_heart: | javac | 2m 35s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 12m 56s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 39m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux d74d63eda987 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5cea8112fd |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 86 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1549080381
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 1s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 4m 22s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 50s | the patch passed |
| +1 :green_heart: | compile | 0m 53s | the patch passed |
| +1 :green_heart: | javac | 0m 53s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 221m 35s | hbase-server in the patch passed. |
| | | 245m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux d1ecf4fd9d1e 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 93ddd7060d |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/testReport/ |
| Max. process+thread count | 4165 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1322937168
@taklwu sir. Thanks for your review. Please wait for my new commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1362869377
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 41s | master passed |
| +1 :green_heart: | compile | 2m 34s | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | master passed |
| +1 :green_heart: | spotless | 0m 38s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 38s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | the patch passed |
| +1 :green_heart: | compile | 2m 32s | the patch passed |
| +1 :green_heart: | javac | 2m 32s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 35s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 8m 41s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. |
| +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 32m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux 059008a19545 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dcfde79f7e |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 79 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1344240711
@taklwu Could you take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "2005hithlj (via GitHub)" <gi...@apache.org>.
2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1057235274
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
return getStoreFileAgeStream().average();
}
+ @Override
+ public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+ Collection<HStoreFile> storeFiles = this.storeEngine.getStoreFileManager().getStorefiles();
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+ for (HStoreFile sf : storeFiles) {
+ if (sf.getReader() == null) {
+ continue;
+ }
+ FileStatus fileStatus;
+ try {
+ fileStatus = sf.getFileInfo().getFileStatus();
Review Comment:
@Apache9 sir.
Yes, this is really a serious problem.
Can we solve this problem by periodically accessing NN randomly, for example, accessing it every hour or more to get fileStatus, so as to reduce the pressure on the NN?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
+ for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+ Long accessTime = pair.getFirst();
+ Long size = pair.getSecond();
+ for (int threshold : storeFilesAccessedDaysThresholds) {
+ Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+ if ((now - accessTime) >= threshold * ONE_DAY_MS) {
Review Comment:
@Apache9 sir.
Yes, this is by design. For instance, if an HFile has not been accessed for 90 days, it is certain that it has also not been accessed for 30 and 7 days.
And what is your suggestion?
##########
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java:
##########
@@ -56,6 +56,8 @@ public interface MetricsRegionSource extends Comparable<MetricsRegionSource> {
String ROW_READS_ONLY_ON_MEMSTORE_DESC = "Row reads happening completely out of memstore";
String MIXED_ROW_READS = "mixedRowReadsCount";
String MIXED_ROW_READS_ON_STORE_DESC = "Row reads happening out of files and memstore on store";
+ String STOREFILES_ACCESSED_DAYS_AND_SIZE = "storeFilesAccessed%sDaysSize";
Review Comment:
OK, sir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1030012124
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -59,15 +61,28 @@ public class MetricsRegionWrapperImpl implements MetricsRegionWrapper, Closeable
private long maxCompactionQueueSize;
private Map<String, Long> readsOnlyFromMemstore;
private Map<String, Long> mixedReadsOnStore;
+ private Map<Integer, Long> storeFilesAccessedDaysAndSize;
+ private int[] storeFilesAccessedDaysThresholds;
private ScheduledFuture<?> regionMetricsUpdateTask;
+ public static long ONE_DAY_MS = 24 * 3600 * 1000;
+ public static final String STOREFILES_ACCESSED_DAYS_THRESHOLDS =
+ "hbase.region.storefiles.accessed.days.thresholds";
+ public static final int[] STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT = { 7, 30, 90 };
Review Comment:
Yeah, in this situation, the metrics will retain at the time the store file flushes or opened for cache, although the cached data of store file is accessed at a new time. I'm not sure whether it is critical issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1055424594
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
return getStoreFileAgeStream().average();
}
+ @Override
+ public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+ Collection<HStoreFile> storeFiles = this.storeEngine.getStoreFileManager().getStorefiles();
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+ for (HStoreFile sf : storeFiles) {
+ if (sf.getReader() == null) {
+ continue;
+ }
+ FileStatus fileStatus;
+ try {
+ fileStatus = sf.getFileInfo().getFileStatus();
Review Comment:
> If this FileStatus is cached, then the accessTime will not be updated?
@Apache9 sir. Thanks for your review.
Yes, if FileStatus is cached, the accessTime will not be updated, but the current logic is got from the back-end file system in real time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1363198753
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 44s | master passed |
| +1 :green_heart: | compile | 0m 57s | master passed |
| +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 46s | the patch passed |
| +1 :green_heart: | compile | 0m 57s | the patch passed |
| +1 :green_heart: | javac | 0m 57s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 24s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 41s | hbase-hadoop-compat in the patch passed. |
| -1 :x: | unit | 260m 36s | hbase-server in the patch failed. |
| | | 286m 11s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 089c6f96ec85 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dcfde79f7e |
| Default Java | Temurin-1.8.0_352-b08 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/testReport/ |
| Max. process+thread count | 2441 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1331764972
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 25s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 3m 47s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 11s | the patch passed |
| +1 :green_heart: | compile | 0m 54s | the patch passed |
| +1 :green_heart: | javac | 0m 54s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 47s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 216m 14s | hbase-server in the patch passed. |
| | | 238m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 178686b71dc4 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3ffc0b5552 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/testReport/ |
| Max. process+thread count | 2482 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache9 commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1055030620
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -313,9 +341,24 @@ public void run() {
tempVal += store.getMemstoreOnlyRowReadsCount();
}
readsOnlyFromMemstore.put(store.getColumnFamilyName(), tempVal);
+
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
Review Comment:
Use `long` instead of `Long` as later we will do calculation?
##########
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java:
##########
@@ -56,6 +56,8 @@ public interface MetricsRegionSource extends Comparable<MetricsRegionSource> {
String ROW_READS_ONLY_ON_MEMSTORE_DESC = "Row reads happening completely out of memstore";
String MIXED_ROW_READS = "mixedRowReadsCount";
String MIXED_ROW_READS_ON_STORE_DESC = "Row reads happening out of files and memstore on store";
+ String STOREFILES_ACCESSED_DAYS_AND_SIZE = "storeFilesAccessed%sDaysSize";
Review Comment:
Better name it XXX_TEMPLATE.
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
return getStoreFileAgeStream().average();
}
+ @Override
+ public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+ Collection<HStoreFile> storeFiles = this.storeEngine.getStoreFileManager().getStorefiles();
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+ for (HStoreFile sf : storeFiles) {
+ if (sf.getReader() == null) {
+ continue;
+ }
+ FileStatus fileStatus;
+ try {
+ fileStatus = sf.getFileInfo().getFileStatus();
Review Comment:
If this FileStatus is cached, then the accessTime will not be updated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1364411372
The failed UTs seems to have nothing to do with my modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1544026821
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 36s | master passed |
| +1 :green_heart: | compile | 0m 53s | master passed |
| +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 40s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | hbase-hadoop-compat in the patch passed. |
| -1 :x: | unit | 203m 38s | hbase-server in the patch failed. |
| | | 228m 26s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux e0452d8fde45 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 5cea8112fd |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/testReport/ |
| Max. process+thread count | 2653 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/4/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1193961037
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
+ for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+ Long accessTime = pair.getFirst();
+ Long size = pair.getSecond();
+ for (int threshold : storeFilesAccessedDaysThresholds) {
+ Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+ if ((now - accessTime) >= threshold * ONE_DAY_MS) {
Review Comment:
So here if the accessTime is very old, we will increase the sumSize for all the thresholds? Is this by design? Seems a bit strange...
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
Review Comment:
Just use long instead of Long?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1548920691
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 25s | master passed |
| +1 :green_heart: | compile | 2m 42s | master passed |
| +1 :green_heart: | checkstyle | 0m 40s | master passed |
| +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 1m 45s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 22s | the patch passed |
| +1 :green_heart: | compile | 2m 42s | the patch passed |
| +1 :green_heart: | javac | 2m 42s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 41s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 13m 14s | Patch does not cause any errors with Hadoop 3.2.4 3.3.5. |
| +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. |
| +1 :green_heart: | spotbugs | 2m 5s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 40m 50s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile |
| uname | Linux b422b054b1d8 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 93ddd7060d |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Max. process+thread count | 85 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/5/console |
| versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "2005hithlj (via GitHub)" <gi...@apache.org>.
2005hithlj commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1543848719
> Please fix the conflicts. And how do you plan to reduce the pressure on NN when fetching the accessTime? Or is this not a big problem?
>
> Thanks.
@Apache9 sir.
I believe that the number of hfile will be in a reasonable state during the compaction of HBase. In addition, with periodic (e.g. every hour) access to the underlying file system and random access hfile between different hstores (ensuring no concentrated access), the actual pressure on the NN will not be too great.
I have a new commit. Could you take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "2005hithlj (via GitHub)" <gi...@apache.org>.
2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1194541848
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
// accumulate the count
mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+ Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+ store.getStoreFilesAccessTimeAndSize();
+ Long now = EnvironmentEdgeManager.currentTime();
+ for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+ Long accessTime = pair.getFirst();
+ Long size = pair.getSecond();
+ for (int threshold : storeFilesAccessedDaysThresholds) {
+ Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+ if ((now - accessTime) >= threshold * ONE_DAY_MS) {
Review Comment:
Ok, sir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1363107006
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 38s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 3m 49s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | the patch passed |
| +1 :green_heart: | compile | 0m 51s | the patch passed |
| +1 :green_heart: | javac | 0m 51s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 49s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 197m 42s | hbase-server in the patch passed. |
| | | 219m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 833a781634b5 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / dcfde79f7e |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/testReport/ |
| Max. process+thread count | 2680 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/3/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1500772896
Please fix the conflicts. And how do you plan to reduce the pressure on NN when fetching the accessTime? Or is this not a big problem?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] taklwu commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
taklwu commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1028549700
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionWrapperImpl.java:
##########
@@ -59,15 +61,28 @@ public class MetricsRegionWrapperImpl implements MetricsRegionWrapper, Closeable
private long maxCompactionQueueSize;
private Map<String, Long> readsOnlyFromMemstore;
private Map<String, Long> mixedReadsOnStore;
+ private Map<Integer, Long> storeFilesAccessedDaysAndSize;
+ private int[] storeFilesAccessedDaysThresholds;
private ScheduledFuture<?> regionMetricsUpdateTask;
+ public static long ONE_DAY_MS = 24 * 3600 * 1000;
+ public static final String STOREFILES_ACCESSED_DAYS_THRESHOLDS =
+ "hbase.region.storefiles.accessed.days.thresholds";
+ public static final int[] STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT = { 7, 30, 90 };
Review Comment:
can we add few line of javadoc or some line in README.md how to use this ?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -52,18 +60,30 @@ public MetricsTableWrapperAggregateImpl(final HRegionServer regionServer) {
this.runnable = new TableMetricsWrapperRunnable();
this.tableMetricsUpdateTask =
this.executor.scheduleWithFixedDelay(this.runnable, period, period, TimeUnit.MILLISECONDS);
+
+ storeFilesAccessedDaysThresholds =
+ regionServer.getConfiguration().getInts(STOREFILES_ACCESSED_DAYS_THRESHOLDS);
+ if (storeFilesAccessedDaysThresholds == null || storeFilesAccessedDaysThresholds.length == 0) {
+ storeFilesAccessedDaysThresholds = STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT;
+ }
}
public class TableMetricsWrapperRunnable implements Runnable {
@Override
public void run() {
Map<TableName, MetricsTableValues> localMetricsTableMap = new HashMap<>();
+
Review Comment:
[nit] is this added by spotless?
```suggestion
```
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:
##########
@@ -382,6 +385,63 @@ public void testAppend() throws Exception {
assertCounter("appendNumOps", 73);
}
+ @Test
+ public void testRegionStoreFilesAccessedDaysAndSize() throws Exception {
+ doNPuts(4000, true);
+ admin.flush(tableName);
+ try (RegionLocator locator = connection.getRegionLocator(tableName)) {
+ for (HRegionLocation location : locator.getAllRegionLocations()) {
+ RegionInfo hri = location.getRegion();
+ MetricsRegionAggregateSource agg =
+ rs.getRegion(hri.getRegionName()).getMetrics().getSource().getAggregateSource();
+ String prefix = "namespace_" + NamespaceDescriptor.DEFAULT_NAMESPACE_NAME_STR + "_table_"
+ + tableName.getNameAsString() + "_region_" + hri.getEncodedName() + "_metric_";
+
+ TEST_UTIL.waitFor(60000, 1000, new Waiter.ExplainingPredicate<Exception>() {
+ @Override
+ public boolean evaluate() {
+ return metricsHelper.checkGaugeExists(prefix + "storeFilesAccessed7DaysSize", agg)
+ && metricsHelper.getGaugeLong(prefix + "storeFilesAccessed7DaysSize", agg) > 0;
+ }
+
+ @Override
+ public String explainFailure() {
+ return prefix + "storeFilesAccessed7DaysSize does not exists in region metrics";
+ }
+ });
+
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed7DaysSize", 0, agg);
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed30DaysSize", 0, agg);
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed90DaysSize", 0, agg);
+ }
+ }
+ }
+
+ @Test
+ public void testTableStoreFilesAccessedDaysAndSize() throws Exception {
+ doNPuts(4000, true);
+ admin.flush(tableName);
+ MetricsTableAggregateSource agg = metricsRegionServer.getTableSourceAgg();
+ String prefix = "namespace_" + NamespaceDescriptor.DEFAULT_NAMESPACE_NAME_STR + "_table_"
+ + tableName.getNameAsString() + "_metric_";
+ TEST_UTIL.waitFor(60000, 1000, new Waiter.ExplainingPredicate<Exception>() {
+ @Override
+ public boolean evaluate() {
+ return metricsHelper.checkGaugeExists(prefix + "storeFilesAccessed7DaysSize", agg)
+ && metricsHelper.getGaugeLong(prefix + "storeFilesAccessed7DaysSize", agg) > 0;
+ }
+
+ @Override
+ public String explainFailure() {
+ return prefix + "storeFilesAccessed7DaysSize does not exists in table metrics";
+ }
+ });
+
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed7DaysSize", 0, agg);
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed30DaysSize", 0, agg);
+ metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed90DaysSize", 0, agg);
Review Comment:
[nit] is there any way to verify non-zero metric for `storeFilesAccessed%sDaysSize` in test, e.g. if we can mock the access time ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1322684271
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 20s | master passed |
| +1 :green_heart: | compile | 0m 53s | master passed |
| +1 :green_heart: | shadedjars | 3m 48s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | the patch passed |
| +1 :green_heart: | compile | 0m 53s | the patch passed |
| +1 :green_heart: | javac | 0m 53s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 45s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 222m 32s | hbase-server in the patch passed. |
| | | 244m 7s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 943f47dee13f 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9462933179 |
| Default Java | Temurin-1.8.0_352-b08 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/testReport/ |
| Max. process+thread count | 2562 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/1/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on PR #4891:
URL: https://github.com/apache/hbase/pull/4891#issuecomment-1331746527
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 40s | master passed |
| +1 :green_heart: | compile | 0m 52s | master passed |
| +1 :green_heart: | shadedjars | 4m 1s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | master passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 52s | the patch passed |
| +1 :green_heart: | javac | 0m 52s | the patch passed |
| +1 :green_heart: | shadedjars | 3m 53s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | hbase-hadoop-compat in the patch passed. |
| +1 :green_heart: | unit | 194m 7s | hbase-server in the patch passed. |
| | | 215m 32s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/4891 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 2b41eb130602 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 3ffc0b5552 |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/testReport/ |
| Max. process+thread count | 2706 (vs. ulimit of 30000) |
| modules | C: hbase-hadoop-compat hbase-server U: . |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4891/2/console |
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org