You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/26 09:30:26 UTC

[GitHub] [hudi] zhangyue19921010 opened a new pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

zhangyue19921010 opened a new pull request #3719:
URL: https://github.com/apache/hudi/pull/3719


   
   ## What is the purpose of the pull request
   Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests
   Cache HoodieTableFileSystemView at baseDir level
   The same as HoodieTableMetaClient
   
   Here is the test result based on S3 hudi table
   240 partitions and 2400 data files.
   
   
   
   Condition | All Requests | Get Requests | List Requests | 4xx errors
   -- | -- | -- | -- | --
   Disable metadata table | 17280 | 14144 | 1322 | 1568
   Disable metadata table + this patch tuning | 11495 | 9839 | 796 | 620
   
   
   
   
   Condition | All Requests | Get Requests | List Requests | 4xx errors
   -- | -- | -- | -- | --
   Enable metadata table | 21024 | 13980 | 3198 | 3438
   Enable metadata table + this patch tuning | 13399 | 10485 | 1093 | 1298
   
   
   
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-942021887


   > @zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC @xushiyan
   
   @nsivabalan Thanks a lot for your review. Yep, I use a query with meta-data enabled at query side to do the benchmark.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718137137



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       it is in flink incremental read code path. still a problem?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4d2fc3d664279975d143494274b787c6a6d5db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401) 
   * 82b6fa38d0f2e8fca6a7804a550b89a4328644f2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r724726381



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());
+          if (null == fsView) {
+            fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext, metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+            hoodieTableFileSystemViewCache.put(baseDir.toString(), fsView);

Review comment:
       Sure thing. Changed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan merged pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3719:
URL: https://github.com/apache/hudi/pull/3719


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718207481



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       `fsView.close()` will do 
   ```
     @Override
     public void close() {
       closed = true;
       super.reset();
       partitionToFileGroupsMap = null;
       fgIdToPendingCompaction = null;
       fgIdToBootstrapBaseFile = null;
       fgIdToReplaceInstants = null;
     }
   
   ```
   
   
   Because of we recycling fsView, we can't close it. Although we only create one fsView for each baseDir. And will cause no memory leak maybe.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718207481



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       `fsView.close()` will do 
   ```
     @Override
     public void close() {
       closed = true;
       super.reset();
       partitionToFileGroupsMap = null;
       fgIdToPendingCompaction = null;
       fgIdToBootstrapBaseFile = null;
       fgIdToReplaceInstants = null;
     }
   
   because of we recycling fsView, we can't close it. Although we create fsView for each baseDir. And will cause no memory leak maybe.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r717684115



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       here the `fsView` would never get updated once put into cache, even the writer commits new file, the new file will not appears in the `fsView`, I think it may lead to wrong results in flink streaming read, right?

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       here the `fsView` would never get updated once put into cache, even the writer commits new file, the new file will not appears in the `fsView`, I think it may lead to wrong results in flink streaming read, right? cc @danny0405 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-938516254


   Hi @vinothchandar I noticed that you are the author of `HoodieROTableFilter.java`. Would you mind take a look at your convince?  
   
   Very appreciate it if you could give me a hand :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] danny0405 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718104690



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());
+          if (null == fsView) {
+            fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext, metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+            hoodieTableFileSystemViewCache.put(baseDir.toString(), fsView);

Review comment:
       Use `computeIfAbsent` to simplify the logic.

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       How should the cached views been closed ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718205032



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Thanks for your review. 
   As we can see HoodieROTablePathFilter already **create and cache** `HoodieTableMetaClient` at baseDir level, also `setLoadActiveTimelineOnLoad(true)` which will create an active timeline in singleton mode.
   So that IMO no matter we cache the fsView or not, any new created files will not appear in current hoodieROTablePathFilter.
   
   Now  we cached the fsView using above cached meta client and cached active timeline. Maybe can have no bad effect but can reduce unnecessary init action.
   
   

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       `fsView.close()` will do 
   ```
     @Override
     public void close() {
       closed = true;
       super.reset();
       partitionToFileGroupsMap = null;
       fgIdToPendingCompaction = null;
       fgIdToBootstrapBaseFile = null;
       fgIdToReplaceInstants = null;
     }
   
   because of we recycling fsView, we can't close it. Although we create fsView for each baseDir. And will cause no memory leak maybe.
   ```

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       `fsView.close()` will do 
   ```
     @Override
     public void close() {
       closed = true;
       super.reset();
       partitionToFileGroupsMap = null;
       fgIdToPendingCompaction = null;
       fgIdToBootstrapBaseFile = null;
       fgIdToReplaceInstants = null;
     }
   
   ```
   
   
   Because of we recycling fsView, we can't close it. Although we only create one fsView for each baseDir. And will cause no memory leak maybe.

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Feel free to correct me if I am wrong :) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r717684115



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       here the `fsView` would never get updated once put into cache, even the writer commits new file, the new file will not appears in the `fsView`, I think it may lead to wrong results in flink streaming read, right? cc @danny0405 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r717684115



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       here the `fsView` would never get updated once put into cache, even the writer commits new file, the new file will not appears in the `fsView`, I think it may lead to wrong results in flink streaming read, right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927578493






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-942021887


   > @zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC @xushiyan
   
   @nsivabalan Thanks a lot for your review. Yep, I use a query with meta-data enabled at query side to do the benchmark.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4d2fc3d664279975d143494274b787c6a6d5db1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718137137



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       it is in flink incremental read code path. still a problem?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2537",
       "triggerID" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4d2fc3d664279975d143494274b787c6a6d5db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401) 
   * 82b6fa38d0f2e8fca6a7804a550b89a4328644f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2537) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718205032



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Thanks for your review. 
   As we can see HoodieROTablePathFilter already **create and cache** `HoodieTableMetaClient` at baseDir level, also `setLoadActiveTimelineOnLoad(true)` which will create an active timeline in singleton mode.
   So that IMO no matter we cache the fsView or not, any new created files will not appear in current hoodieROTablePathFilter.
   
   Now  we cached the fsView using above cached meta client and cached active timeline. Maybe can have no bad effect but can reduce unnecessary init action.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] danny0405 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718104393



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Flink streaming reader does not use this code path.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4d2fc3d664279975d143494274b787c6a6d5db1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927311933


   I am a little curious about why the requests in `Enable metadata table` is larger than `Disable metadata table`, and also what the query response time difference between enable/disable metadata table?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-941259434


   @zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC  @xushiyan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2537",
       "triggerID" : "82b6fa38d0f2e8fca6a7804a550b89a4328644f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 82b6fa38d0f2e8fca6a7804a550b89a4328644f2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2537) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927578493






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] danny0405 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718104393



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Flink streaming reader does not use this code path.

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());
+          if (null == fsView) {
+            fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext, metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+            hoodieTableFileSystemViewCache.put(baseDir.toString(), fsView);

Review comment:
       Use `computeIfAbsent` to simplify the logic.

##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -202,10 +212,6 @@ public boolean accept(Path path) {
           }
           nonHoodiePathCache.add(folder.toString());
           return true;
-        } finally {
-          if (fsView != null) {
-            fsView.close();
-          }

Review comment:
       How should the cached views been closed ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-941259434


   @zhangyue19921010 : wrt your comment on perf difference (enabling and disabling metadata), was it a read query that you benchmarked? If yes, did you also enable metadata when you queried? CC  @xushiyan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#discussion_r718316505



##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java
##########
@@ -175,8 +181,12 @@ public boolean accept(Path path) {
             metaClientCache.put(baseDir.toString(), metaClient);
           }
 
-          fsView = FileSystemViewManager.createInMemoryFileSystemView(engineContext,
-              metaClient, HoodieInputFormatUtils.buildMetadataConfig(getConf()));
+          fsView = hoodieTableFileSystemViewCache.get(baseDir.toString());

Review comment:
       Feel free to correct me if I am wrong :) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3719: [HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTableFileSystemView, aiming to reduce unnecessary list/get requests

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3719:
URL: https://github.com/apache/hudi/pull/3719#issuecomment-927270024


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401",
       "triggerID" : "f4d2fc3d664279975d143494274b787c6a6d5db1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f4d2fc3d664279975d143494274b787c6a6d5db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2401) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org