You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2025/12/11 06:01:12 UTC

[PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

CTTY opened a new pull request, #17560:
URL: https://github.com/apache/hudi/pull/17560

   ### Describe the issue this Pull Request addresses
   
   <!-- Either describe the issue inline here with motivation behind the changes 
        (or) link to an issue by including `Closes #<issue-number>` for context. 
        If this PR includes changes to the storage format, public APIs,
        or has breaking changes, use `!` (e.g., feat!: ...) -->
   Use StorageUtils.getStorage rather than using constructors directly
   ### Summary and Changelog
   
   <!-- Short, plain-English summary of what users gain or what changed in behavior.
        Followed by a detailed log of all the changes. Highlight if any code was copied. -->
   - Use StorageUtils.getStorage rather than using HoodieHadoopStorage constructor directly in multiple places
   ### Impact
   
   <!-- Describe any public API or user-facing feature change or any performance impact. -->
   None
   ### Risk Level
   none
   <!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
        If medium or high, explain what verification was done to mitigate the risks. -->
   
   ### Documentation Update
   none
   <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
   
   - The config description must be updated if new configs are added or the default value of the configs are changed.
   - Any new feature or user-facing change requires updating the Hudi website. Please follow the 
     [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3648153318

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10182",
       "triggerID" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10182) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3640340207

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4d1c6ecf7fc561e28d45368a28c8f95567fbdad6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3640347786

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4d1c6ecf7fc561e28d45368a28c8f95567fbdad6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3640616082

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4d1c6ecf7fc561e28d45368a28c8f95567fbdad6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17560:
URL: https://github.com/apache/hudi/pull/17560


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17560:
URL: https://github.com/apache/hudi/pull/17560#discussion_r2613124565


##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java:
##########
@@ -526,8 +524,8 @@ public static String getTableBasePath(InputSplit split, JobConf jobConf) throws
       return realtimeSplit.getBasePath();
     } else {
       Path inputPath = ((FileSplit) split).getPath();
-      FileSystem fs = inputPath.getFileSystem(jobConf);
-      HoodieStorage storage = new HoodieHadoopStorage(fs);
+      HoodieStorage storage = HoodieStorageUtils.getStorage(
+              new StoragePath(inputPath.toUri()), HadoopFSUtils.getStorageConf(jobConf));

Review Comment:
   nit: use `HadoopFSUtils.convertToStoragePath`



##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java:
##########
@@ -312,7 +313,7 @@ public static Schema addPartitionFields(Schema schema, List<String> partitioning
   public static HoodieFileReader getBaseFileReader(Path path, JobConf conf) throws IOException {
     StorageConfiguration<?> storageConf = HadoopFSUtils.getStorageConf(conf);
     HoodieConfig hoodieConfig = getReaderConfigs(storageConf);
-    return HoodieIOFactory.getIOFactory(new HoodieHadoopStorage(path, conf))
+    return HoodieIOFactory.getIOFactory(HoodieStorageUtils.getStorage(new StoragePath(path.toUri()), storageConf))

Review Comment:
   Same here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3647701328

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10182",
       "triggerID" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4d1c6ecf7fc561e28d45368a28c8f95567fbdad6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122) 
   * 5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10182) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Avoid using HoodieHadoopStorage directly [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17560:
URL: https://github.com/apache/hudi/pull/17560#issuecomment-3647695656

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122",
       "triggerID" : "4d1c6ecf7fc561e28d45368a28c8f95567fbdad6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4d1c6ecf7fc561e28d45368a28c8f95567fbdad6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10122) 
   * 5d0881d6c9bfcc1ce3d5fdb01b0f0bfc7218ffdf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org