You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bhasudha (via GitHub)" <gi...@apache.org> on 2023/08/20 10:22:25 UTC
[GitHub] [hudi] bhasudha opened a new pull request, #9486: [DOCS] Update Key generation page
bhasudha opened a new pull request, #9486:
URL: https://github.com/apache/hudi/pull/9486
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1321557338
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
+| hoodie.datasource.write.hive_style_partitioning | false (Optional) | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /><br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+| hoodie.datasource.write.partitionpath.urlencode | false (Optional) | Should we url encode the partition path value, before creating the folder structure.<br /><br />`Config Param: URL_ENCODE_PARTITIONING` |
-There are few more configs involved if you are looking for TimestampBasedKeyGenerator. Will cover those in the respective section.
+For all advanced configs refer [here](https://hudi.apache.org/docs/next/configurations#KEY_GENERATOR).
Lets go over different key generators available to be used with Hudi.
### [SimpleKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SimpleKeyGenerator.java)
Record key refers to one field(column in dataframe) by name and partition path refers to one field (single column in dataframe)
-by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
+by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
### [ComplexKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java)
Both record key and partition paths comprise one or more than one field by name(combination of multiple fields). Fields
are expected to be comma separated in the config value. For example ```"Hoodie.datasource.write.recordkey.field" : “col1,col4”```
-### [GlobalDeleteKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/GlobalDeleteKeyGenerator.java)
-Global index deletes do not require partition value. So this key generator avoids using partition value for generating HoodieKey.
-
-**_NOTE:_**
-The "GlobalDeleteKeyGenerator" has to be used with a global index to delete records solely based on the record key.
-It works for a batch with deletes only. The key generator can be used for both partitioned and non-partitioned table. Note that
-when using this key generator, the config `hoodie.[bloom|simple|hbase].index.update.partition.path` should be set to
-`false` in order to avoid redundant data written to the storage.
-
### [NonpartitionedKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/NonpartitionedKeyGenerator.java)
If your hudi dataset is not partitioned, you could use this “NonpartitionedKeyGenerator” which will return an empty
partition for all records. In other words, all records go to the same partition (which is empty “”)
+### [AutoRecordGenWrapperKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/AutoRecordGenWrapperKeyGenerator.java)
Review Comment:
okay sounds good. Will remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1299351791
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
+| hoodie.datasource.write.hive_style_partitioning | false (Optional) | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /><br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+| hoodie.datasource.write.partitionpath.urlencode | false (Optional) | Should we url encode the partition path value, before creating the folder structure.<br /><br />`Config Param: URL_ENCODE_PARTITIONING` |
-There are few more configs involved if you are looking for TimestampBasedKeyGenerator. Will cover those in the respective section.
+For all advanced configs refer [here](https://hudi.apache.org/docs/next/configurations#KEY_GENERATOR).
Lets go over different key generators available to be used with Hudi.
### [SimpleKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SimpleKeyGenerator.java)
Record key refers to one field(column in dataframe) by name and partition path refers to one field (single column in dataframe)
-by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
+by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
### [ComplexKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java)
Both record key and partition paths comprise one or more than one field by name(combination of multiple fields). Fields
are expected to be comma separated in the config value. For example ```"Hoodie.datasource.write.recordkey.field" : “col1,col4”```
-### [GlobalDeleteKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/GlobalDeleteKeyGenerator.java)
-Global index deletes do not require partition value. So this key generator avoids using partition value for generating HoodieKey.
-
-**_NOTE:_**
-The "GlobalDeleteKeyGenerator" has to be used with a global index to delete records solely based on the record key.
-It works for a batch with deletes only. The key generator can be used for both partitioned and non-partitioned table. Note that
-when using this key generator, the config `hoodie.[bloom|simple|hbase].index.update.partition.path` should be set to
-`false` in order to avoid redundant data written to the storage.
-
### [NonpartitionedKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/NonpartitionedKeyGenerator.java)
If your hudi dataset is not partitioned, you could use this “NonpartitionedKeyGenerator” which will return an empty
partition for all records. In other words, all records go to the same partition (which is empty “”)
+### [AutoRecordGenWrapperKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/AutoRecordGenWrapperKeyGenerator.java)
Review Comment:
@nsivabalan please review and suggest edits on this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1321561227
##########
website/docs/key_generation.md:
##########
@@ -7,9 +7,13 @@ last_modified_at:
Every record in Hudi is uniquely identified by a primary key, which is a pair of record key and partition path where the record belongs to.
Using primary keys, Hudi can impose a) partition level uniqueness integrity constraint b) enable fast updates and deletes on records.
One should choose the partitioning scheme wisely as it could be a determining factor for your ingestion and query latency.
+Some use cases do not have a naturally present record key, for ex. log ingestion type of payloads. For these type of use cases,
Review Comment:
Done. Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1299351584
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
Review Comment:
@nsivabalan please review and suggest edits on this part. I have edited it to cover for the auto key generation part as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #9486: [DOCS] Update Key generation page
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #9486:
URL: https://github.com/apache/hudi/pull/9486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1322192804
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
hudi will auto infer partition path.
for eg, if not configured, hudi will assume its non-partitioned.
So, the key gen class is not strictly a mandatory field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1321566891
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
Review Comment:
`When not configured record key will be automatically generated by Hudi.` is already there. I extended it to add `When not configured record key will be automatically generated by Hudi. This feature is handy for use cases like log ingestion that do not have a naturally present record key.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1299351718
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
@nsivabalan please review and suggest edits on this part. I have edited it to cover for the auto key generation and auto inferencing part as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on PR #9486:
URL: https://github.com/apache/hudi/pull/9486#issuecomment-1685248556
Tested locally!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1316487787
##########
website/docs/key_generation.md:
##########
@@ -7,9 +7,13 @@ last_modified_at:
Every record in Hudi is uniquely identified by a primary key, which is a pair of record key and partition path where the record belongs to.
Using primary keys, Hudi can impose a) partition level uniqueness integrity constraint b) enable fast updates and deletes on records.
One should choose the partitioning scheme wisely as it could be a determining factor for your ingestion and query latency.
+Some use cases do not have a naturally present record key, for ex. log ingestion type of payloads. For these type of use cases,
Review Comment:
also, can we add somewhere that auto record key generation is from 0.14.0 onwards.
##########
website/docs/key_generation.md:
##########
@@ -7,9 +7,13 @@ last_modified_at:
Every record in Hudi is uniquely identified by a primary key, which is a pair of record key and partition path where the record belongs to.
Using primary keys, Hudi can impose a) partition level uniqueness integrity constraint b) enable fast updates and deletes on records.
One should choose the partitioning scheme wisely as it could be a determining factor for your ingestion and query latency.
+Some use cases do not have a naturally present record key, for ex. log ingestion type of payloads. For these type of use cases,
Review Comment:
looks good.
minor suggestion
```
compute, storage and read to meet the uniqueness requirements
```
"and" -> "to"
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
Review Comment:
looks good.
in the end, we can add something like
"When this field is omitted, hudi auto generates the record key for every record. for most immutable use-cases users may not need to configure a record key".
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
looks good.
minor suggestion
```
In case of auto generated record keys, the configure key generator class/type will be used to determine the partition path generation. So, users can still configure a key generator class and skip configuring record key field (on which case, hudi will auto generate record keys).
```
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
+| hoodie.datasource.write.hive_style_partitioning | false (Optional) | Flag to indicate whether to use Hive style partitioning. If set true, the names of partition folders follow <partition_column_name>=<partition_value> format. By default false (the names of partition folders are only partition values)<br /><br />`Config Param: HIVE_STYLE_PARTITIONING_ENABLE` |
+| hoodie.datasource.write.partitionpath.urlencode | false (Optional) | Should we url encode the partition path value, before creating the folder structure.<br /><br />`Config Param: URL_ENCODE_PARTITIONING` |
-There are few more configs involved if you are looking for TimestampBasedKeyGenerator. Will cover those in the respective section.
+For all advanced configs refer [here](https://hudi.apache.org/docs/next/configurations#KEY_GENERATOR).
Lets go over different key generators available to be used with Hudi.
### [SimpleKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SimpleKeyGenerator.java)
Record key refers to one field(column in dataframe) by name and partition path refers to one field (single column in dataframe)
-by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
+by name. This is one of the most commonly used one. Values are interpreted as is from dataframe and converted to string.
### [ComplexKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/ComplexKeyGenerator.java)
Both record key and partition paths comprise one or more than one field by name(combination of multiple fields). Fields
are expected to be comma separated in the config value. For example ```"Hoodie.datasource.write.recordkey.field" : “col1,col4”```
-### [GlobalDeleteKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/GlobalDeleteKeyGenerator.java)
-Global index deletes do not require partition value. So this key generator avoids using partition value for generating HoodieKey.
-
-**_NOTE:_**
-The "GlobalDeleteKeyGenerator" has to be used with a global index to delete records solely based on the record key.
-It works for a batch with deletes only. The key generator can be used for both partitioned and non-partitioned table. Note that
-when using this key generator, the config `hoodie.[bloom|simple|hbase].index.update.partition.path` should be set to
-`false` in order to avoid redundant data written to the storage.
-
### [NonpartitionedKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/NonpartitionedKeyGenerator.java)
If your hudi dataset is not partitioned, you could use this “NonpartitionedKeyGenerator” which will return an empty
partition for all records. In other words, all records go to the same partition (which is empty “”)
+### [AutoRecordGenWrapperKeyGenerator](https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/AutoRecordGenWrapperKeyGenerator.java)
Review Comment:
I would suggest to remove this from here. This is not supposed to be used by anyone.
might confuse users whether we expect them to set this key gen class so that hudi will auto generate record keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1322222145
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
Thanks for clarifying. Edited the smae. ptal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1299351718
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
@nsivabalan please review and suggest edits on this part. I have edited it to cover for the auto key generation part as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1299351447
##########
website/docs/key_generation.md:
##########
@@ -7,9 +7,13 @@ last_modified_at:
Every record in Hudi is uniquely identified by a primary key, which is a pair of record key and partition path where the record belongs to.
Using primary keys, Hudi can impose a) partition level uniqueness integrity constraint b) enable fast updates and deletes on records.
One should choose the partitioning scheme wisely as it could be a determining factor for your ingestion and query latency.
+Some use cases do not have a naturally present record key, for ex. log ingestion type of payloads. For these type of use cases,
Review Comment:
@nsivabalan please review and suggest edits on this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bhasudha commented on a diff in pull request #9486: [DOCS] Update Key generation page
Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha commented on code in PR #9486:
URL: https://github.com/apache/hudi/pull/9486#discussion_r1321582193
##########
website/docs/key_generation.md:
##########
@@ -20,43 +24,44 @@ generators that are readily available to use.
[Here](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-common/src/main/java/org/apache/hudi/keygen/KeyGenerator.java)
is the interface for KeyGenerator in Hudi for your reference.
-Before diving into different types of key generators, let’s go over some of the common configs required to be set for
-key generators.
+Before diving into different types of key generators, let’s go over some of the common configs relevant to key generators.
-| Config | Meaning/purpose|
-| ------------- |:-------------:|
-| ```hoodie.datasource.write.recordkey.field``` | Refers to record key field. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.field``` | Refers to partition path field. This is a mandatory field. |
-| ```hoodie.datasource.write.keygenerator.class``` | Refers to Key generator class(including full path). Could refer to any of the available ones or user defined one. This is a mandatory field. |
-| ```hoodie.datasource.write.partitionpath.urlencode```| When set to true, partition path will be url encoded. Default value is false. |
-| ```hoodie.datasource.write.hive_style_partitioning```| When set to true, uses hive style partitioning. Partition field name will be prefixed to the value. Format: “<partition_path_field_name>=<partition_path_value>”. Default value is false.|
+| Config Name | Default | Description |
+| ---------------------------------------------------------------------------------------- |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.datasource.write.recordkey.field | N/A (Optional) | Record key field. Value to be used as the `recordKey` component of `HoodieKey`. <ul><li>When configured, actual value will be obtained by invoking .toString() on the field value. Nested fields can be specified using the dot notation eg: `a.b.c`. </li><li>When not configured record key will be automatically generated by Hudi.</li></ul> <br />`Config Param: RECORDKEY_FIELD_NAME` |
+| hoodie.datasource.write.partitionpath.field | N/A (Optional) | Partition path field. Value to be used at the partitionPath component of HoodieKey. This needs to be specified if a partitioned table is desired. Actual value obtained by invoking .toString()<br />`Config Param: PARTITIONPATH_FIELD_NAME` |
+| hoodie.datasource.write.keygenerator.class | N/A (Optional) | Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. <ul><li>When set, the configured value takes precedence to be in effect and automatic inference is not triggered.</li><li>When not configured, if `hoodie.datasource.write.keygenerator.type` is set, the configured value is used else automatic inference is triggered.</li><li>In case of auto generated record keys. the key generator type is automatically inferred.</li></ul> <br />`Config Param: KEYGENERATOR_CLASS_NAME` |
Review Comment:
@nsivabalan For specifying partition path, the users should still configure the keygen class/type correct ? Or is it assumed to be non-partitioned if the key gen class is not provided ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org