You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nfarah86 (via GitHub)" <gi...@apache.org> on 2024/02/13 01:35:30 UTC
[PR] Docs clarification around read and write indexes [hudi]
nfarah86 opened a new pull request, #10658:
URL: https://github.com/apache/hudi/pull/10658
updated the indexing docs so the record index includes point looks up & also did some minor clean up on the writing of the other indexes
<img width="1011" alt="Screenshot 2024-02-12 at 5 33 44 PM" src="https://github.com/apache/hudi/assets/5392555/56558e00-eaaa-4550-ba44-9922d593362c">
@xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs clarification around read and write indexes [hudi]
Posted by "nfarah86 (via GitHub)" <gi...@apache.org>.
nfarah86 commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503411628
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
Review Comment:
@xushiyan are you referring to global_simple description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS][MINOR] Docs clarification around read and write indexes [hudi]
Posted by "nfarah86 (via GitHub)" <gi...@apache.org>.
nfarah86 commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503591192
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,18 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
-- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced across all partitions in the table.
-- **HBASE:** Manages the index mapping in an external Apache HBase table.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. It requires keys to be partition-level or table/global-level unique so it can function correctly.
Review Comment:
I understand- the indexes should be updated now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs clarification around read and write indexes [hudi]
Posted by "nfarah86 (via GitHub)" <gi...@apache.org>.
nfarah86 commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503403885
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
Review Comment:
thanks @xushiyan will re-word.
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
-- **HBASE:** Manages the index mapping in an external Apache HBase table.
+- **HBASE:** Oversees the index mapping through an external table in Apache HBase.
Review Comment:
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs clarification around read and write indexes [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503079397
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
-- **HBASE:** Manages the index mapping in an external Apache HBase table.
+- **HBASE:** Oversees the index mapping through an external table in Apache HBase.
- **INMEMORY (default for Flink and Java):** Uses in-memory hashmap in Spark and Java engine and Flink in-memory state in Flink for indexing.
-- **BUCKET:** Employs bucket hashing to locates the file group containing the records. Particularly beneficial in
- large scale. Use `hoodie.index.bucket.engine` to choose bucket engine type, i.e., how buckets are generated;
- - `SIMPLE(default)`: Uses a fixed number of buckets for file groups per partition which cannot shrink or expand. This works for both COW and
- MOR tables. Since the num of buckets cannot be changed and design of one-on-one mapping between buckets and file groups,
- this index might not suit well for highly skewed partitions.
- - `CONSISTENT_HASHING`: Supports dynamic number of buckets with bucket resizing to properly size each bucket. This
- solves potential data skew problem where partitions with high volume of data can be dynamically resized to have
- multiple buckets that are reasonably sized in contrast to the fixed number of buckets per partition in SIMPLE
- bucket engine type. This only works with MOR tables.
-- **RECORD_INDEX:** Index which saves the record key to location mappings in the HUDI Metadata Table. Record index is a
- global index, enforcing key uniqueness across all partitions in the table. Supports sharding to achieve very high scale.
+- **BUCKET:** Utilizes bucket hashing to identify the file group that houses the records, which proves to be particularly advantageous on a large scale. To select the type of bucket engine—that is, the method by which buckets are created—use the `hoodie.index.bucket.engine` configuration option.
+ - `SIMPLE(default)`: This index employs a fixed number of buckets for file groups within each partition, which do not have the capacity to decrease or increase in size. It is applicable to both COW and MOR tables. Due to the unchangeable number of buckets and the design principle of mapping each bucket to a single file group, this indexing method may not be ideal for partitions with significant data skew.
+ - `CONSISTENT_HASHING`: This index accommodates a dynamic number of buckets, with the capability for bucket resizing to ensure each bucket is sized appropriately. This addresses the issue of data skew in partitions with a high volume of data by allowing these partitions to be dynamically resized. As a result, partitions can have multiple reasonably sized buckets, unlike the fixed bucket count per partition seen in the SIMPLE bucket engine type. This feature is exclusively compatible with MOR tables.
+
+- **RECORD_INDEX:** This index saves the record key to location mappings in the HUDI Metadata Table. It functions as a global index, ensuring that keys are unique across all partitions within the table. To accommodate very high scales, it utilizes sharding. The record index is specifically optimized to enable fast upserts. Additionally, when it comes to reading data, the index is crafted to allow for rapid point lookups, significantly speeding up data retrieval processes.
Review Comment:
same here. it needs key to be unique to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs clarification around read and write indexes [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503076148
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
Review Comment:
there is a misconception: index does not ensure key uniqueness, it requires keys to be partition-level or table/global-level unique so it can function correctly. we should phrase it that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS][MINOR] Docs clarification around read and write indexes [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503470026
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,18 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
-- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced across all partitions in the table.
-- **HBASE:** Manages the index mapping in an external Apache HBase table.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. It requires keys to be partition-level or table/global-level unique so it can function correctly.
Review Comment:
"partition-level or table/global-level unique" this needs fix case by case, as in non-global index requires partition level uniqueness, and global index requires table-level uniqueness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs clarification around read and write indexes [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #10658:
URL: https://github.com/apache/hudi/pull/10658#discussion_r1503077755
##########
website/docs/indexing.md:
##########
@@ -27,27 +27,19 @@ could end up having to merge all the base files against all incoming updates/del
Currently, Hudi supports the following index types. Default is SIMPLE on Spark engine, and INMEMORY on Flink and Java
engines.
-- **BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges.Key uniqueness is enforced inside partitions.
-- **GLOBAL_BLOOM:** Employs bloom filters built out of the record keys, optionally also pruning candidate files using
- record key ranges. Key uniqueness is enforced across all partitions in the table.
-- **SIMPLE (default for Spark engines):** Default index type for spark engine. Performs a lean join of the incoming records against keys extracted from the table on
- storage. Key uniqueness is enforced inside partitions.
+- **BLOOM:** Uses bloom filters generated from record keys, with the option to further narrow down candidate files based on the ranges of the record keys. This method ensures the uniqueness of keys within partitions.
+- **GLOBAL_BLOOM:** Utilizes bloom filters created from record keys, and may also refine the selection of candidate files by using the ranges of record keys. This ensures that keys are unique throughout all partitions in the table.
+- **SIMPLE (default for Spark engines):** This is the standard index type for the Spark engine. It executes an efficient join of incoming records with keys retrieved from the table stored on disk, ensuring that keys are unique within partitions.
- **GLOBAL_SIMPLE:** Performs a lean join of the incoming records against keys extracted from the table on
storage. Key uniqueness is enforced across all partitions in the table.
-- **HBASE:** Manages the index mapping in an external Apache HBase table.
+- **HBASE:** Oversees the index mapping through an external table in Apache HBase.
Review Comment:
"oversee" is a bit odd here. i think the original is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS][MINOR] Docs clarification around read and write indexes [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #10658:
URL: https://github.com/apache/hudi/pull/10658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org