You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:46:43 UTC

[I] [RFC-60] Optimized storage layout for cloud object stores [hudi]

hudi-bot opened a new issue, #15062:
URL: https://github.com/apache/hudi/issues/15062

   Amazon S3 among other cloud object stores, throttle requests based on object prefix => [https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/]. Hudi follows the traditional Hive storage layout, with files being stored under separate partition paths under a common table path/prefix. This introduces the potential for throttling because of request limits being reached for the common table path/prefix, when writing significant number of files concurrently.
   
   We propose implementing an alternate storage layout, that would be more suitable for cloud object stores like S3 to avoid running into throttling issues as the data scales. At a high level, we need to be able to distribute data files evenly across randomly generated prefixes, so that request limits get distributed across those prefixes, instead of a single table prefix.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-3625
   - Type: Epic
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [RFC-60] Optimized storage layout for cloud object stores [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15062: [RFC-60] Optimized storage layout for cloud object stores
URL: https://github.com/apache/hudi/issues/15062


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [RFC-60] Optimized storage layout for cloud object stores [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15062:
URL: https://github.com/apache/hudi/issues/15062#issuecomment-3630113534

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/5113
   - https://github.com/apache/hudi/pull/6440
   - https://github.com/apache/hudi/pull/6882
   - https://github.com/apache/hudi/pull/9462
   - https://github.com/apache/hudi/pull/11607
   - https://github.com/apache/hudi/pull/12407
   - https://github.com/apache/hudi/pull/12460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [RFC-60] Optimized storage layout for cloud object stores [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15062:
URL: https://github.com/apache/hudi/issues/15062#issuecomment-4795307630

   Closing as resolved. Tracked in [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) (fixVersions: 1.1.0). The change landed in PR [#9462](https://github.com/apache/hudi/pull/9462) (commit `849b217edb56`). Please reopen if you can still reproduce on the latest release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org