You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/19 14:08:33 UTC
[GitHub] [hudi] wzx140 opened a new pull request, #5637: [HUDI-4124] Add valid check in Spark Datasource configs
wzx140 opened a new pull request, #5637:
URL: https://github.com/apache/hudi/pull/5637
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Add valid check in Spark Datasource configs
## Brief change log
- make keyGenerator default value more reasonable in Spark Datasource configs
- add valid check in Spark Datasource configs
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138409615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943",
"triggerID" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"triggerType" : "PUSH"
}, {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943",
"triggerID" : "1138290153",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 224ab27b3b758171cd86f3377d9eba327e9b5820 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r878133161
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java:
##########
@@ -57,7 +64,11 @@ public class HoodieIndexConfig extends HoodieConfig {
public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
.key("hoodie.index.type")
- .noDefaultValue()
+ .defaultValue(BLOOM.name())
Review Comment:
can we make it `SIMPLE` here? we switched to SIMPLE default here https://github.com/apache/hudi/pull/4659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1132905652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc838867d8ee1c7332b79c11cc1e0f62614ee975 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769)
* 010c94ed70bff816fd674c753e0763a0307ffdff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1131929701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc838867d8ee1c7332b79c11cc1e0f62614ee975 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wzx140 commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
wzx140 commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882358849
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigProperty.java:
##########
@@ -95,33 +99,46 @@ Option<Function<HoodieConfig, Option<T>>> getInferFunc() {
return inferFunction;
}
+ public void checkValues(String value) {
+ if (validValues != null && !validValues.isEmpty() && !validValues.contains(value)) {
+ throw new IllegalArgumentException(
+ "The value of " + key + " should be one of "
+ + String.join(",", validValues) + ", but was " + value);
+ }
+ }
+
public List<String> getAlternatives() {
return Arrays.asList(alternatives);
}
public ConfigProperty<T> withDocumentation(String doc) {
Objects.requireNonNull(doc);
- return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, alternatives);
+ return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, validValues, alternatives);
+ }
+
+ public ConfigProperty<T> withValidValues(List<String> validValues) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138404839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 224ab27b3b758171cd86f3377d9eba327e9b5820 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882357471
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java:
##########
@@ -57,7 +64,11 @@ public class HoodieIndexConfig extends HoodieConfig {
public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
.key("hoodie.index.type")
- .noDefaultValue()
+ .defaultValue(SIMPLE.name())
Review Comment:
@wzx140 i now think it should be better to leave it `noDefaultValue()` given it'll be controlled by `getDefaultIndexType()` from hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138402510
CI passed in https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138312009
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1138290153",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138407297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943",
"triggerID" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"triggerType" : "PUSH"
}, {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1138290153",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 224ab27b3b758171cd86f3377d9eba327e9b5820 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882337916
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigProperty.java:
##########
@@ -95,33 +99,46 @@ Option<Function<HoodieConfig, Option<T>>> getInferFunc() {
return inferFunction;
}
+ public void checkValues(String value) {
+ if (validValues != null && !validValues.isEmpty() && !validValues.contains(value)) {
+ throw new IllegalArgumentException(
+ "The value of " + key + " should be one of "
+ + String.join(",", validValues) + ", but was " + value);
+ }
+ }
+
public List<String> getAlternatives() {
return Arrays.asList(alternatives);
}
public ConfigProperty<T> withDocumentation(String doc) {
Objects.requireNonNull(doc);
- return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, alternatives);
+ return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, validValues, alternatives);
+ }
+
+ public ConfigProperty<T> withValidValues(List<String> validValues) {
Review Comment:
Why List? Why not Set?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java:
##########
@@ -57,7 +64,11 @@ public class HoodieIndexConfig extends HoodieConfig {
public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
.key("hoodie.index.type")
- .noDefaultValue()
+ .defaultValue(SIMPLE.name())
Review Comment:
There is already `HoodieIndexConfig#getDefaultIndexType` method which sets default which gets called by the builder. That method sets value based on execution engine. Why are we setting default here? I am not sure simple index (which relies on spark lean join) would work with flink. cc @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wzx140 commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
wzx140 commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882359777
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java:
##########
@@ -57,7 +64,11 @@ public class HoodieIndexConfig extends HoodieConfig {
public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
.key("hoodie.index.type")
- .noDefaultValue()
+ .defaultValue(SIMPLE.name())
Review Comment:
@xushiyan ok. I leave it noDefaultValue() and add comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wzx140 commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
wzx140 commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138290153
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #5637:
URL: https://github.com/apache/hudi/pull/5637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138224620
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 010c94ed70bff816fd674c753e0763a0307ffdff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787)
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138314602
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "1138290153",
"triggerType" : "MANUAL"
}, {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941)
* 224ab27b3b758171cd86f3377d9eba327e9b5820 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138264601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1131780490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc838867d8ee1c7332b79c11cc1e0f62614ee975 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1131774587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc838867d8ee1c7332b79c11cc1e0f62614ee975 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wzx140 commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
wzx140 commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882456452
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigProperty.java:
##########
@@ -95,33 +99,46 @@ Option<Function<HoodieConfig, Option<T>>> getInferFunc() {
return inferFunction;
}
+ public void checkValues(String value) {
+ if (validValues != null && !validValues.isEmpty() && !validValues.contains(value)) {
+ throw new IllegalArgumentException(
+ "The value of " + key + " should be one of "
+ + String.join(",", validValues) + ", but was " + value);
+ }
+ }
+
public List<String> getAlternatives() {
return Arrays.asList(alternatives);
}
public ConfigProperty<T> withDocumentation(String doc) {
Objects.requireNonNull(doc);
- return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, alternatives);
+ return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, validValues, alternatives);
+ }
+
+ public ConfigProperty<T> withValidValues(List<String> validValues) {
Review Comment:
@codope Fixed. Use `withValidValues(String... validValues)` and store them with set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882439372
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigProperty.java:
##########
@@ -95,33 +99,46 @@ Option<Function<HoodieConfig, Option<T>>> getInferFunc() {
return inferFunction;
}
+ public void checkValues(String value) {
+ if (validValues != null && !validValues.isEmpty() && !validValues.contains(value)) {
+ throw new IllegalArgumentException(
+ "The value of " + key + " should be one of "
+ + String.join(",", validValues) + ", but was " + value);
+ }
+ }
+
public List<String> getAlternatives() {
return Arrays.asList(alternatives);
}
public ConfigProperty<T> withDocumentation(String doc) {
Objects.requireNonNull(doc);
- return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, alternatives);
+ return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, validValues, alternatives);
+ }
+
+ public ConfigProperty<T> withValidValues(List<String> validValues) {
Review Comment:
From that perspective, I agree list is more consistent. Perf does not matter much for just a few elements.
@wzx140 sorry for the trouble, but can you revert it back to `withValidValues(String... validValues)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wzx140 commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
wzx140 commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r878144831
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java:
##########
@@ -57,7 +64,11 @@ public class HoodieIndexConfig extends HoodieConfig {
public static final ConfigProperty<String> INDEX_TYPE = ConfigProperty
.key("hoodie.index.type")
- .noDefaultValue()
+ .defaultValue(BLOOM.name())
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1133036982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 010c94ed70bff816fd674c753e0763a0307ffdff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1132909629
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dc838867d8ee1c7332b79c11cc1e0f62614ee975 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769)
* 010c94ed70bff816fd674c753e0763a0307ffdff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138317161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941",
"triggerID" : "1138290153",
"triggerType" : "MANUAL"
}, {
"hash" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943",
"triggerID" : "224ab27b3b758171cd86f3377d9eba327e9b5820",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8941)
* 224ab27b3b758171cd86f3377d9eba327e9b5820 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5637:
URL: https://github.com/apache/hudi/pull/5637#discussion_r882358860
##########
hudi-common/src/main/java/org/apache/hudi/common/config/ConfigProperty.java:
##########
@@ -95,33 +99,46 @@ Option<Function<HoodieConfig, Option<T>>> getInferFunc() {
return inferFunction;
}
+ public void checkValues(String value) {
+ if (validValues != null && !validValues.isEmpty() && !validValues.contains(value)) {
+ throw new IllegalArgumentException(
+ "The value of " + key + " should be one of "
+ + String.join(",", validValues) + ", but was " + value);
+ }
+ }
+
public List<String> getAlternatives() {
return Arrays.asList(alternatives);
}
public ConfigProperty<T> withDocumentation(String doc) {
Objects.requireNonNull(doc);
- return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, alternatives);
+ return new ConfigProperty<>(key, defaultValue, doc, sinceVersion, deprecatedVersion, inferFunction, validValues, alternatives);
+ }
+
+ public ConfigProperty<T> withValidValues(List<String> validValues) {
Review Comment:
Guess this is not for performance concern, and more for easier API usage. varargs actually works better. see the other API `withAlternatives(String... alternatives)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5637: [HUDI-4124] Add valid check in Spark Datasource configs
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5637:
URL: https://github.com/apache/hudi/pull/5637#issuecomment-1138222194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8769",
"triggerID" : "dc838867d8ee1c7332b79c11cc1e0f62614ee975",
"triggerType" : "PUSH"
}, {
"hash" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787",
"triggerID" : "010c94ed70bff816fd674c753e0763a0307ffdff",
"triggerType" : "PUSH"
}, {
"hash" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22c9e8779f0ff4c31441eb58a58bd69b33c04d77",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 010c94ed70bff816fd674c753e0763a0307ffdff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8787)
* 22c9e8779f0ff4c31441eb58a58bd69b33c04d77 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org