You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/23 00:03:05 UTC
[GitHub] [hudi] kywe665 opened a new pull request #4076: [HUDI-2822] Doc for auto file sizing guide
kywe665 opened a new pull request #4076:
URL: https://github.com/apache/hudi/pull/4076
## What is the purpose of the pull request
Added docs for auto file sizing management guide.
## Brief change log
- Added file-sizing doc
- Added it to the guides section
## Verify this pull request
This is a docs change only
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [X] Necessary doc changes done or have another open PR
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kywe665 commented on pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
kywe665 commented on pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#issuecomment-978121227
we need to land cleaner PR first: #4052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#discussion_r754882176
##########
File path: website/docs/file_sizing.md
##########
@@ -0,0 +1,53 @@
+---
+title: "Auto File Size Management"
+toc: true
+---
+
+This doc will show you how Apache Hudi overcomes the dreaded small files problem. A key design decision in Hudi was to
+avoid creating small files in the first place and always write properly sized files.
+There are 2 ways to manage small files in Hudi and below will describe the advantages and trade-offs of each.
+
+## Auto-Size During ingestion
+
+You can automatically manage size of files during ingestion. This solution adds a little latency during ingestion, but
+it ensures that read queries are always efficient as soon as a write is committed. If you don't
+manage file sizing as you write and instead try to periodically run a file-sizing clean-up, your queries will be slow until that resize cleanup is periodically performed.
+
+(Note: [bulk_insert](/docs/next/write_operations) write operation does not provide auto-sizing during ingestion)
+
+### For Copy-On-Write
+This is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize)
+and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should
+be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
+ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
+record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
+configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+files < 100MB and try to get them upto 120MB.
+
+### For Merge-On-Read
+MergeOnRead works differently for different INDEX choices so there are few more configs to set:
+
+- Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can
+configure the [maximum log size](/docs/configurations#hoodielogfilemaxsize) and a
Review comment:
(/docs/configurations/#hoodielogfilemaxsize)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#discussion_r754875730
##########
File path: website/docs/file_sizing.md
##########
@@ -0,0 +1,53 @@
+---
+title: "Auto File Size Management"
+toc: true
+---
+
+This doc will show you how Apache Hudi overcomes the dreaded small files problem. A key design decision in Hudi was to
+avoid creating small files in the first place and always write properly sized files.
+There are 2 ways to manage small files in Hudi and below will describe the advantages and trade-offs of each.
+
+## Auto-Size During ingestion
+
+You can automatically manage size of files during ingestion. This solution adds a little latency during ingestion, but
+it ensures that read queries are always efficient as soon as a write is committed. If you don't
+manage file sizing as you write and instead try to periodically run a file-sizing clean-up, your queries will be slow until that resize cleanup is periodically performed.
+
+(Note: [bulk_insert](/docs/next/write_operations) write operation does not provide auto-sizing during ingestion)
Review comment:
The link to bulk_insert is pointing to next/write_operations. Might want to change that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a change in pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
bhasudha commented on a change in pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#discussion_r754876441
##########
File path: website/docs/file_sizing.md
##########
@@ -0,0 +1,53 @@
+---
+title: "Auto File Size Management"
+toc: true
+---
+
+This doc will show you how Apache Hudi overcomes the dreaded small files problem. A key design decision in Hudi was to
+avoid creating small files in the first place and always write properly sized files.
+There are 2 ways to manage small files in Hudi and below will describe the advantages and trade-offs of each.
+
+## Auto-Size During ingestion
+
+You can automatically manage size of files during ingestion. This solution adds a little latency during ingestion, but
+it ensures that read queries are always efficient as soon as a write is committed. If you don't
+manage file sizing as you write and instead try to periodically run a file-sizing clean-up, your queries will be slow until that resize cleanup is periodically performed.
+
+(Note: [bulk_insert](/docs/next/write_operations) write operation does not provide auto-sizing during ingestion)
Review comment:
Also you might want to change `does not provide` to `does not gurantee` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kywe665 commented on a change in pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
kywe665 commented on a change in pull request #4076:
URL: https://github.com/apache/hudi/pull/4076#discussion_r755695067
##########
File path: website/docs/file_sizing.md
##########
@@ -0,0 +1,53 @@
+---
+title: "Auto File Size Management"
+toc: true
+---
+
+This doc will show you how Apache Hudi overcomes the dreaded small files problem. A key design decision in Hudi was to
+avoid creating small files in the first place and always write properly sized files.
+There are 2 ways to manage small files in Hudi and below will describe the advantages and trade-offs of each.
+
+## Auto-Size During ingestion
+
+You can automatically manage size of files during ingestion. This solution adds a little latency during ingestion, but
+it ensures that read queries are always efficient as soon as a write is committed. If you don't
+manage file sizing as you write and instead try to periodically run a file-sizing clean-up, your queries will be slow until that resize cleanup is periodically performed.
+
+(Note: [bulk_insert](/docs/next/write_operations) write operation does not provide auto-sizing during ingestion)
Review comment:
write_operations is a net new doc not found in v0.9.0. Build will break if I point to /docs/write_operations. Since there are so many net new docs, I suggest we leave links as /docs/next/ for now and then after we cut 0.10.0 release this week we can sweep through to remove the /next/?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha merged pull request #4076: [HUDI-2822] Doc for auto file sizing guide
Posted by GitBox <gi...@apache.org>.
bhasudha merged pull request #4076:
URL: https://github.com/apache/hudi/pull/4076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org