You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "PaddyMelody (via GitHub)" <gi...@apache.org> on 2023/04/28 07:45:53 UTC

[GitHub] [hudi] PaddyMelody opened a new pull request, #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

PaddyMelody opened a new pull request, #8595:
URL: https://github.com/apache/hudi/pull/8595

   ### Change Logs
   
   Fixed hadoop configuration not being applied by org.apache.hudi.source.FileIndex
   
   ### Impact
   FileIndex uses the DEFAULT HoodieFlinkEngineContext to get the partitionPath without using the information in the configuration.
   <img width="981" alt="image" src="https://user-images.githubusercontent.com/46181516/235086491-257b22ab-3b7c-4a09-8b5b-983ec2ffccfc.png">
   Since I was connecting to a remote hadoop, I subsequently got the following error due to missing configuration
   <img width="1331" alt="image" src="https://user-images.githubusercontent.com/46181516/235085666-3a0b19f9-7432-480e-9c75-6ad76d8df1c2.png">
   <img width="1385" alt="image" src="https://user-images.githubusercontent.com/46181516/235085696-b021fe53-9d93-4f53-9098-bb720ec97369.png">
   
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184472506


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -66,15 +66,19 @@ public class FileIndex {
   private final RowType rowType;
   private final boolean tableExists;
   private final HoodieMetadataConfig metadataConfig;
+  private final HoodieFlinkEngineContext hoodieFlinkEngineContext;
   private final PartitionPruners.PartitionPruner partitionPruner; // for partition pruning
   private final DataPruner dataPruner;                            // for data skipping
   private final int dataBucket;                                   // for bucket pruning
   private List<String> partitionPaths;                            // cache of partition paths
 
   private FileIndex(Path path, Configuration conf, RowType rowType, DataPruner dataPruner, PartitionPruners.PartitionPruner partitionPruner, int dataBucket) {
+    org.apache.hadoop.conf.Configuration hadoopConf = HadoopConfigurations.getHadoopConf(conf);
+

Review Comment:
   Generating the `HoodieFlinkEngineContext` on the fly should be fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1527184198

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184470886


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -66,15 +66,19 @@ public class FileIndex {
   private final RowType rowType;
   private final boolean tableExists;
   private final HoodieMetadataConfig metadataConfig;
+  private final HoodieFlinkEngineContext hoodieFlinkEngineContext;
   private final PartitionPruners.PartitionPruner partitionPruner; // for partition pruning
   private final DataPruner dataPruner;                            // for data skipping
   private final int dataBucket;                                   // for bucket pruning
   private List<String> partitionPaths;                            // cache of partition paths
 
   private FileIndex(Path path, Configuration conf, RowType rowType, DataPruner dataPruner, PartitionPruners.PartitionPruner partitionPruner, int dataBucket) {
+    org.apache.hadoop.conf.Configuration hadoopConf = HadoopConfigurations.getHadoopConf(conf);
+

Review Comment:
   yeah, I will keep the hadoop conf as member



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1534093550

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     }, {
       "hash" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16812",
       "triggerID" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 21e3090d2bd0eb714322e3ffad7b3554f4440829 UNKNOWN
   * ae38bcb32800fe4f6a14ee1e627607296041c10a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16812) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184454362


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -145,7 +149,7 @@ public FileStatus[] getFilesInPartitions() {
       return new FileStatus[0];
     }
     String[] partitions = getOrBuildPartitionPaths().stream().map(p -> fullPartitionPath(path, p)).toArray(String[]::new);
-    FileStatus[] allFiles = FSUtils.getFilesInPartitions(HoodieFlinkEngineContext.DEFAULT, metadataConfig, path.toString(), partitions)
+    FileStatus[] allFiles = FSUtils.getFilesInPartitions(hoodieFlinkEngineContext, metadataConfig, path.toString(), partitions)

Review Comment:
   Something like that
   <img width="761" alt="image" src="https://user-images.githubusercontent.com/46181516/236088448-e9fdd856-ffc2-4edd-b790-3ea80f4caa96.png">



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184469395


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -66,15 +66,19 @@ public class FileIndex {
   private final RowType rowType;
   private final boolean tableExists;
   private final HoodieMetadataConfig metadataConfig;
+  private final HoodieFlinkEngineContext hoodieFlinkEngineContext;
   private final PartitionPruners.PartitionPruner partitionPruner; // for partition pruning
   private final DataPruner dataPruner;                            // for data skipping
   private final int dataBucket;                                   // for bucket pruning
   private List<String> partitionPaths;                            // cache of partition paths
 
   private FileIndex(Path path, Configuration conf, RowType rowType, DataPruner dataPruner, PartitionPruners.PartitionPruner partitionPruner, int dataBucket) {
+    org.apache.hadoop.conf.Configuration hadoopConf = HadoopConfigurations.getHadoopConf(conf);
+

Review Comment:
   Can we keep the hadoop conf as member instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184472150


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -66,15 +66,19 @@ public class FileIndex {
   private final RowType rowType;
   private final boolean tableExists;
   private final HoodieMetadataConfig metadataConfig;
+  private final HoodieFlinkEngineContext hoodieFlinkEngineContext;
   private final PartitionPruners.PartitionPruner partitionPruner; // for partition pruning
   private final DataPruner dataPruner;                            // for data skipping
   private final int dataBucket;                                   // for bucket pruning
   private List<String> partitionPaths;                            // cache of partition paths
 
   private FileIndex(Path path, Configuration conf, RowType rowType, DataPruner dataPruner, PartitionPruners.PartitionPruner partitionPruner, int dataBucket) {
+    org.apache.hadoop.conf.Configuration hadoopConf = HadoopConfigurations.getHadoopConf(conf);
+

Review Comment:
   I got a warning. Can we ignore it? Or do I need to make any other changes
   <img width="1378" alt="image" src="https://user-images.githubusercontent.com/46181516/236094572-fe2e3206-fea6-4304-b06b-8dfae6cc0366.png">
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184474743


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -66,15 +66,19 @@ public class FileIndex {
   private final RowType rowType;
   private final boolean tableExists;
   private final HoodieMetadataConfig metadataConfig;
+  private final HoodieFlinkEngineContext hoodieFlinkEngineContext;
   private final PartitionPruners.PartitionPruner partitionPruner; // for partition pruning
   private final DataPruner dataPruner;                            // for data skipping
   private final int dataBucket;                                   // for bucket pruning
   private List<String> partitionPaths;                            // cache of partition paths
 
   private FileIndex(Path path, Configuration conf, RowType rowType, DataPruner dataPruner, PartitionPruners.PartitionPruner partitionPruner, int dataBucket) {
+    org.apache.hadoop.conf.Configuration hadoopConf = HadoopConfigurations.getHadoopConf(conf);
+

Review Comment:
   you are right.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1527135803

   PTAL.@danny0405 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1534021944

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     }, {
       "hash" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16812",
       "triggerID" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729) 
   * 21e3090d2bd0eb714322e3ffad7b3554f4440829 UNKNOWN
   * ae38bcb32800fe4f6a14ee1e627607296041c10a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16812) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1534013742

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     }, {
       "hash" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ae38bcb32800fe4f6a14ee1e627607296041c10a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729) 
   * 21e3090d2bd0eb714322e3ffad7b3554f4440829 UNKNOWN
   * ae38bcb32800fe4f6a14ee1e627607296041c10a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] PaddyMelody commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "PaddyMelody (via GitHub)" <gi...@apache.org>.
PaddyMelody commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1184454362


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -145,7 +149,7 @@ public FileStatus[] getFilesInPartitions() {
       return new FileStatus[0];
     }
     String[] partitions = getOrBuildPartitionPaths().stream().map(p -> fullPartitionPath(path, p)).toArray(String[]::new);
-    FileStatus[] allFiles = FSUtils.getFilesInPartitions(HoodieFlinkEngineContext.DEFAULT, metadataConfig, path.toString(), partitions)
+    FileStatus[] allFiles = FSUtils.getFilesInPartitions(hoodieFlinkEngineContext, metadataConfig, path.toString(), partitions)

Review Comment:
   Something like that
   <img width="1032" alt="image" src="https://user-images.githubusercontent.com/46181516/236091497-e2cd91c2-f37d-4a38-a294-59e427a25669.png">
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1527193645

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8595:
URL: https://github.com/apache/hudi/pull/8595#discussion_r1181469799


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -145,7 +149,7 @@ public FileStatus[] getFilesInPartitions() {
       return new FileStatus[0];
     }
     String[] partitions = getOrBuildPartitionPaths().stream().map(p -> fullPartitionPath(path, p)).toArray(String[]::new);
-    FileStatus[] allFiles = FSUtils.getFilesInPartitions(HoodieFlinkEngineContext.DEFAULT, metadataConfig, path.toString(), partitions)
+    FileStatus[] allFiles = FSUtils.getFilesInPartitions(hoodieFlinkEngineContext, metadataConfig, path.toString(), partitions)

Review Comment:
   What kind of hadoop configuration do you wanna to pass around?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1533985956

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     }, {
       "hash" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "21e3090d2bd0eb714322e3ffad7b3554f4440829",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729) 
   * 21e3090d2bd0eb714322e3ffad7b3554f4440829 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8595:
URL: https://github.com/apache/hudi/pull/8595#issuecomment-1527595290

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729",
       "triggerID" : "38e644e9bd5d2dfb345dcbab6b6d4946f5124988",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38e644e9bd5d2dfb345dcbab6b6d4946f5124988 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 merged pull request #8595: [MINOR] Fixed hadoop configuration not being applied by FileIndex

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8595:
URL: https://github.com/apache/hudi/pull/8595


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org