[GitHub] [hudi] bhasudha commented on a change in pull request #4076: [HUDI-2822] Doc for auto file sizing guide

[GitBox <gi...@apache.org>]

bhasudha commented on a change in pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#discussion_r754875730



##########
File path: website/docs/file_sizing.md
##########
@@ -0,0 +1,53 @@
+---
+title: "Auto File Size Management"
+toc: true
+---
+
+This doc will show you how Apache Hudi overcomes the dreaded small files problem. A key design decision in Hudi was to 
+avoid creating small files in the first place and always write properly sized files. 
+There are 2 ways to manage small files in Hudi and below will describe the advantages and trade-offs of each.
+
+## Auto-Size During ingestion
+
+You can automatically manage size of files during ingestion. This solution adds a little latency during ingestion, but
+it ensures that read queries are always efficient as soon as a write is committed. If you don't 
+manage file sizing as you write and instead try to periodically run a file-sizing clean-up, your queries will be slow until that resize cleanup is periodically performed.
+ 
+(Note: [bulk_insert](/docs/next/write_operations) write operation does not provide auto-sizing during ingestion)

Review comment:
       The link to bulk_insert is pointing to next/write_operations. Might want to change that ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org