[GitHub] [hbase] 2005hithlj commented on a diff in pull request #4891: HBASE-27483 Expose table and region storefiles accessed days and size to the metrics

["2005hithlj (via GitHub)" <gi...@apache.org>]

2005hithlj commented on code in PR #4891:
URL: https://github.com/apache/hbase/pull/4891#discussion_r1057235274


##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java:
##########
@@ -1780,6 +1781,29 @@ public OptionalDouble getAvgStoreFileAge() {
     return getStoreFileAgeStream().average();
   }
 
+  @Override
+  public Map<String, Pair<Long, Long>> getStoreFilesAccessTimeAndSize() {
+    Collection<HStoreFile> storeFiles = this.storeEngine.getStoreFileManager().getStorefiles();
+    Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = new HashMap<>();
+    for (HStoreFile sf : storeFiles) {
+      if (sf.getReader() == null) {
+        continue;
+      }
+      FileStatus fileStatus;
+      try {
+        fileStatus = sf.getFileInfo().getFileStatus();

Review Comment:
   @Apache9  sir.
   Yes, this is really a serious problem. 
   Can we solve this problem by periodically accessing NN randomly, for example, accessing it every hour or more to get  fileStatus, so as to reduce the pressure on the NN?



##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java:
##########
@@ -115,10 +134,23 @@ public void run() {
             // accumulate the count
             mt.perStoreMemstoreOnlyReadCount.put(tempKey, memstoreReadCount);
             mt.perStoreMixedReadCount.put(tempKey, mixedReadCount);
+            Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap =
+              store.getStoreFilesAccessTimeAndSize();
+            Long now = EnvironmentEdgeManager.currentTime();
+            for (Pair<Long, Long> pair : sfAccessTimeAndSizeMap.values()) {
+              Long accessTime = pair.getFirst();
+              Long size = pair.getSecond();
+              for (int threshold : storeFilesAccessedDaysThresholds) {
+                Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold);
+                if ((now - accessTime) >= threshold * ONE_DAY_MS) {

Review Comment:
   @Apache9 sir.
   Yes, this is by design. For instance, if an HFile has not been accessed for 90 days, it is certain that it has also not been accessed for 30 and 7 days. 
   And what is your suggestion?



##########
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java:
##########
@@ -56,6 +56,8 @@ public interface MetricsRegionSource extends Comparable<MetricsRegionSource> {
   String ROW_READS_ONLY_ON_MEMSTORE_DESC = "Row reads happening completely out of memstore";
   String MIXED_ROW_READS = "mixedRowReadsCount";
   String MIXED_ROW_READS_ON_STORE_DESC = "Row reads happening out of files and memstore on store";
+  String STOREFILES_ACCESSED_DAYS_AND_SIZE = "storeFilesAccessed%sDaysSize";

Review Comment:
   OK, sir



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org