You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/04/24 06:30:30 UTC
[GitHub] [hudi] yihua opened a new pull request, #8557: [HUDI-5895] Remove bootstrap key generator configs
yihua opened a new pull request, #8557:
URL: https://github.com/apache/hudi/pull/8557
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1521621184
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79402c2cb4e8482b02cd8aaad613774703ceb3d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537579606
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186752046
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -591,8 +591,7 @@ class TestHoodieSparkSqlWriter {
HoodieBootstrapConfig.PARALLELISM_VALUE.key -> "4",
DataSourceWriteOptions.OPERATION.key -> DataSourceWriteOptions.BOOTSTRAP_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
- HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key -> classOf[NonpartitionedKeyGenerator].getCanonicalName)
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "")
Review Comment:
After this change, we have to set the partition path to empty String. Otherwise, the following error is thrown:
```
org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
PartitionPath: partition
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186751959
##########
hudi-examples/hudi-examples-spark/src/main/java/org/apache/hudi/examples/spark/HoodieSparkBootstrapExample.java:
##########
@@ -64,7 +64,6 @@ public static void main(String[] args) throws Exception {
.option(DataSourceWriteOptions.PRECOMBINE_FIELD().key(), preCombineField)
.option(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieFileFormat.ORC.name())
.option(HoodieBootstrapConfig.BASE_PATH.key(), basePath)
- .option(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(), NonpartitionedKeyGenerator.class.getCanonicalName())
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1519529295
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1519518838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #8557:
URL: https://github.com/apache/hudi/pull/8557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537239942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 75d4c5b67703349573569677d88deb2b1eb647fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717)
* 2679efffa32d3f41bc7d80b9377b8267e5bdc3db Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1529787180
@yihua : there are some CI failures and GH action failures. can you take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1176094157
##########
hudi-examples/hudi-examples-spark/src/main/java/org/apache/hudi/examples/spark/HoodieSparkBootstrapExample.java:
##########
@@ -64,7 +64,6 @@ public static void main(String[] args) throws Exception {
.option(DataSourceWriteOptions.PRECOMBINE_FIELD().key(), preCombineField)
.option(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieFileFormat.ORC.name())
.option(HoodieBootstrapConfig.BASE_PATH.key(), basePath)
- .option(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(), NonpartitionedKeyGenerator.class.getCanonicalName())
Review Comment:
shouldn't we replace w/ data source key gen class config ?
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -263,10 +262,9 @@ private void initializeTable() throws IOException {
private Pair<String, String> genKeyGenClassAndPartitionColumns() {
String keyGenClass;
- if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(), null))) {
- keyGenClass = props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key());
- } else if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key(), null))) {
- props.put(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key()));
+ if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(), null))) {
Review Comment:
how does backwards compatability work? we are not going to support older configs? and ask users to set write configs instead?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -591,8 +591,7 @@ class TestHoodieSparkSqlWriter {
HoodieBootstrapConfig.PARALLELISM_VALUE.key -> "4",
DataSourceWriteOptions.OPERATION.key -> DataSourceWriteOptions.BOOTSTRAP_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
- HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key -> classOf[NonpartitionedKeyGenerator].getCanonicalName)
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "")
Review Comment:
shouldn't we set the key gen class instead of setting partition path config to empty string ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537882236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
}, {
"hash" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16935",
"triggerID" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
* 4c0819e67974da3d144922e3556e6bfed6aeb05c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16935)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1538191091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
}, {
"hash" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16935",
"triggerID" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c0819e67974da3d144922e3556e6bfed6aeb05c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16935)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186754142
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -411,6 +411,11 @@ object DataSourceWriteOptions {
.markAdvanced()
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`")
+ val KEYGENERATOR_TYPE: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.keygenerator.type")
+ .defaultValue(KeyGeneratorType.SIMPLE.name)
+ .withDocumentation("Key generator type.")
Review Comment:
we might need to add more docs here. for eg, which one takes precedence among the key gen class and key gen type. thats why we removed it sometime back.
whats the real need to add it back ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186752072
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -591,8 +591,7 @@ class TestHoodieSparkSqlWriter {
HoodieBootstrapConfig.PARALLELISM_VALUE.key -> "4",
DataSourceWriteOptions.OPERATION.key -> DataSourceWriteOptions.BOOTSTRAP_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
- HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key -> classOf[NonpartitionedKeyGenerator].getCanonicalName)
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "")
Review Comment:
Key gen class name config is kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537268636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2679efffa32d3f41bc7d80b9377b8267e5bdc3db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1521174962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600)
* a79402c2cb4e8482b02cd8aaad613774703ceb3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537601707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537872717
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
}, {
"hash" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4c0819e67974da3d144922e3556e6bfed6aeb05c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
* 4c0819e67974da3d144922e3556e6bfed6aeb05c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537302369
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
}, {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2679efffa32d3f41bc7d80b9377b8267e5bdc3db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903)
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186752187
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -263,10 +262,9 @@ private void initializeTable() throws IOException {
private Pair<String, String> genKeyGenClassAndPartitionColumns() {
String keyGenClass;
- if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(), null))) {
- keyGenClass = props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key());
- } else if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key(), null))) {
- props.put(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key()));
+ if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(), null))) {
Review Comment:
Since the bootstrap operation is an one-time operation, it is ok to ask user to use new configs and we'll mention in the docs that the old configs are removed. Also, in common use cases where simple keygen is used, such a config is not required.
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -263,10 +262,9 @@ private void initializeTable() throws IOException {
private Pair<String, String> genKeyGenClassAndPartitionColumns() {
String keyGenClass;
- if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(), null))) {
- keyGenClass = props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key());
- } else if (StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key(), null))) {
- props.put(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key()));
+ if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(), null))) {
Review Comment:
Since the bootstrap operation is a one-time operation, it is ok to ask the user to use new configs and we'll mention in the docs that the old configs are removed. Also, in common use cases where simple keygen is used, such a config is not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186784885
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -411,6 +411,11 @@ object DataSourceWriteOptions {
.markAdvanced()
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`")
+ val KEYGENERATOR_TYPE: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.keygenerator.type")
+ .defaultValue(KeyGeneratorType.SIMPLE.name)
+ .withDocumentation("Key generator type.")
Review Comment:
Sg. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537365496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
}, {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537238922
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 75d4c5b67703349573569677d88deb2b1eb647fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717)
* 2679efffa32d3f41bc7d80b9377b8267e5bdc3db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1520034171
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537577012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537312173
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
}, {
"hash" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903",
"triggerID" : "2679efffa32d3f41bc7d80b9377b8267e5bdc3db",
"triggerType" : "PUSH"
}, {
"hash" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910",
"triggerID" : "36ccd4fddc346692d11803f5673c60d4e5811dc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2679efffa32d3f41bc7d80b9377b8267e5bdc3db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16903)
* 36ccd4fddc346692d11803f5673c60d4e5811dc6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16910)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1538459165
CI failure is irrelevant to the changes. Merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1526934343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 75d4c5b67703349573569677d88deb2b1eb647fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1521179818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600)
* a79402c2cb4e8482b02cd8aaad613774703ceb3d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1526858818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79402c2cb4e8482b02cd8aaad613774703ceb3d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632)
* 75d4c5b67703349573569677d88deb2b1eb647fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1526863201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16600",
"triggerID" : "7bfc0a7c79ea3adc4d35690ca88cd6e56fb9b75a",
"triggerType" : "PUSH"
}, {
"hash" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632",
"triggerID" : "a79402c2cb4e8482b02cd8aaad613774703ceb3d",
"triggerType" : "PUSH"
}, {
"hash" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717",
"triggerID" : "75d4c5b67703349573569677d88deb2b1eb647fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79402c2cb4e8482b02cd8aaad613774703ceb3d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16632)
* 75d4c5b67703349573569677d88deb2b1eb647fd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16717)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8557:
URL: https://github.com/apache/hudi/pull/8557#issuecomment-1537232463
> @yihua : there are some CI failures and GH action failures. can you take a look
The test failures are fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186771703
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -411,6 +411,11 @@ object DataSourceWriteOptions {
.markAdvanced()
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`")
+ val KEYGENERATOR_TYPE: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.keygenerator.type")
+ .defaultValue(KeyGeneratorType.SIMPLE.name)
+ .withDocumentation("Key generator type.")
Review Comment:
This is to support bootstrap keygen type. I can remove it if it's not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8557: [HUDI-5895] Remove bootstrap key generator configs
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1186777965
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -411,6 +411,11 @@ object DataSourceWriteOptions {
.markAdvanced()
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator`")
+ val KEYGENERATOR_TYPE: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.keygenerator.type")
+ .defaultValue(KeyGeneratorType.SIMPLE.name)
+ .withDocumentation("Key generator type.")
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org