You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/29 10:30:35 UTC
[GitHub] [hudi] flashJd opened a new pull request, #6246: Deepnova
flashJd opened a new pull request, #6246:
URL: https://github.com/apache/hudi/pull/6246
## What is the purpose of the pull request
when migrate external data to hudi, we extract the schema of the source table and use it to create hudi table.
but when the schema contains a column named ts(a common situation), hudi will always precombine data using this column and we can't disable it.
## Brief change log
Be able to disable precombine field when table schema contains a field named ts
## Verify this pull request
modify a test: testRequiredOptionsForSource in TestHoodieTableFactory.java
add a test: testWriteAndReadWithProctimeSequenceWithTsColumnExisting in ITTestHoodieDataSource.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #6246: [HUDI-4543] Support natural order when table schema contains a field named 'ts'
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6246:
URL: https://github.com/apache/hudi/pull/6246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1203544590
@danny0405 can you review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937271200
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
> Is this change valid ? We still set up the option if it is already `FlinkOptions.NO_PRE_COMBINE`.
@danny0405 looking forward to your reply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: [HUDI-4543] able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1206029611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
}, {
"hash" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10592",
"triggerID" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b04e73fecb574e199a3aad9e74dd6c9ae45d123 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205121023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c6cfaaeb0512d426753e989b6fcc72c5d79293b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455)
* 39773f8cf8f7a8441963080fd43d8ea04f1a74c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1199131031
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c6cfaaeb0512d426753e989b6fcc72c5d79293b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205125856
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c6cfaaeb0512d426753e989b6fcc72c5d79293b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455)
* 39773f8cf8f7a8441963080fd43d8ea04f1a74c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: [HUDI-4543] able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205965800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
}, {
"hash" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10592",
"triggerID" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39773f8cf8f7a8441963080fd43d8ea04f1a74c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586)
* 7b04e73fecb574e199a3aad9e74dd6c9ae45d123 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10592)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #6246: [HUDI-4543] Support natural order when table schema contains a field named 'ts'
Posted by GitBox <gi...@apache.org>.
flashJd commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1210067571
> > Thanks for the contribution, can we log a JIRA issue and change the commit titile to form like: `[HUDI-${JIRA issue ID}] ${your actual commit title}`.
>
> I've filed a Jira ticket and changed the commit titile, also fixed the checkStyle conflicts, can you help approval the workflow, thanks.
@danny0405 can we merge it to the master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937547056
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
Then we can do such logic:
else if (!preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)) {
throw new HoodieValidationException("Field " + preCombineField + " does not exist in the table schema."
+ "Please check '" + FlinkOptions.PRECOMBINE_FIELD.key() + "' option.");
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: [HUDI-4543] able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205962815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
}, {
"hash" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b04e73fecb574e199a3aad9e74dd6c9ae45d123",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39773f8cf8f7a8441963080fd43d8ea04f1a74c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586)
* 7b04e73fecb574e199a3aad9e74dd6c9ae45d123 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937318329
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
One thing need to note here is that: option `FlinkOptions.NO_PRE_COMBINE` is only for internal use.
Still curious about your use case, do you mean you want the proc_time/natural order sequence semantics even though you got a `ts` field in the schema ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937468508
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
yes, i want the proc_time/natural order sequence semantics even though you got a ts field in the schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1199125029
@dannyhchen can you review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937636228
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
yes, it is more elegant. I've modified to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #6246: [HUDI-4543] able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205972885
> Thanks for the contribution, can we log a JIRA issue and change the commit titile to form like: `[HUDI-${JIRA issue ID}] ${your actual commit title}`.
I've filed a Jira ticket and changed the commit titile, also fixed the checkStyle conflicts, can you help approval the workflow, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1199228989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c6cfaaeb0512d426753e989b6fcc72c5d79293b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r937318329
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
One thing need to note here is that: option `FlinkOptions.NO_PRE_COMBINE` is only for internal use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205263065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10455",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
}, {
"hash" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586",
"triggerID" : "39773f8cf8f7a8441963080fd43d8ea04f1a74c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39773f8cf8f7a8441963080fd43d8ea04f1a74c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10586)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r936432853
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
it's valid, suppose a table with a 'ts' column, we want to set PRECOMBINE_FIELD=FlinkOptions.NO_PRE_COMBINE to disable the precombine field, above code will go into line 153, because preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) is false, FlinkOptions.PRECOMBINE_FIELD.defaultValue() is 'ts', preCombineField is what we set 'no_precombine', you can also see the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] flashJd commented on pull request #6246: [HUDI-4543] able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
flashJd commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205972641
>
I've filed a Jira ticket and changed the commit titile, also fixed the checkStyle conflicts, can you help approval the workflow, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1205197490
Thanks for the contribution, can we log a JIRA issue and change the commit titile to form like:
`[HUDI-${JIRA issue ID}] ${your actual commit title}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6246:
URL: https://github.com/apache/hudi/pull/6246#issuecomment-1199127280
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c6cfaaeb0512d426753e989b6fcc72c5d79293b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c6cfaaeb0512d426753e989b6fcc72c5d79293b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6246: Be able to disable precombine field when table schema contains a field named ts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6246:
URL: https://github.com/apache/hudi/pull/6246#discussion_r936416513
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java:
##########
@@ -147,7 +147,7 @@ private void sanityCheck(Configuration conf, ResolvedSchema schema) {
throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
+ "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
}
- if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue())) {
+ if (preCombineField.equals(FlinkOptions.PRECOMBINE_FIELD.defaultValue()) || preCombineField.equals(FlinkOptions.NO_PRE_COMBINE)){
conf.setString(FlinkOptions.PRECOMBINE_FIELD, FlinkOptions.NO_PRE_COMBINE);
Review Comment:
Is this change valid ? We still set up the option if it is already `FlinkOptions.NO_PRE_COMBINE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org