You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2024/12/09 19:38:18 UTC
[PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
yihua opened a new pull request, #12452:
URL: https://github.com/apache/hudi/pull/12452
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12452:
URL: https://github.com/apache/hudi/pull/12452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529229951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ec55f23e0ed3d9f17f4981276d818b0fda5723d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926526952
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java:
##########
@@ -264,12 +270,27 @@ public UnaryOperator<ArrayWritable> projectRecord(Schema from, Schema to, Map<St
}
@Override
- public Comparable castValue(Comparable value, Schema.Type newType) {
- //TODO: [HUDI-8261] actually do casting here
+ public Comparable convertValueToEngineType(Comparable value) {
if (value instanceof WritableComparable) {
return value;
}
- return (WritableComparable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(value, Schema.create(newType));
+ //TODO: [HUDI-8261] cover more types
+ if (value == null) {
+ return null;
+ } else if (value instanceof String) {
+ return new Text((String) value);
+ } else if (value instanceof Integer) {
+ return new IntWritable((int) value);
+ } else if (value instanceof Long) {
+ return new LongWritable((long) value);
+ } else if (value instanceof Float) {
+ return new FloatWritable((float) value);
+ } else if (value instanceof Double) {
+ return new DoubleWritable((double) value);
+ } else if (value instanceof Boolean) {
Review Comment:
It should work as we use `#compareTo` to compare the ordering value of particular types. And `(integer) 0` is handled separately and is not used to compared with actual ordering value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2530080387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10f082f6ed1b23f9357e0d5d833653516d5f1145 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345)
* e5f919ce39525f4ca5a7f94fce87a99b0caea430 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926515303
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
I think `COMMIT_TIME_ORDERING_VALUE` is a better name since it indicates commit time ordering. `DEFAULT` or `NON_EXISTANT` might not be obvious, and may lead to confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1928093289
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
In Hudi 1.x, for `OverwriteWithLatestAvroPayload` the ordering value should not be used. We can discuss what's a better name for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529723542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* deec65c355f2cc65234f00cfb86cec18cdc6de98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342)
* 10f082f6ed1b23f9357e0d5d833653516d5f1145 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529721430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* deec65c355f2cc65234f00cfb86cec18cdc6de98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342)
* 10f082f6ed1b23f9357e0d5d833653516d5f1145 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926524919
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -116,9 +115,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
+ ? Option.empty()
+ : Option.ofNullable(ConfigUtils.getOrderingField(props))
+ .or(() -> {
+ String preCombineField = hoodieTableMetaClient.getTableConfig().getPreCombineField();
+ if (StringUtils.isNullOrEmpty(preCombineField)) {
Review Comment:
`Option.ofNullable()` does not handle empty `String` so that's why I keep this logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529911490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10f082f6ed1b23f9357e0d5d833653516d5f1145 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529477992
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ec55f23e0ed3d9f17f4981276d818b0fda5723d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340)
* deec65c355f2cc65234f00cfb86cec18cdc6de98 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926515303
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
I think `COMMIT_TIME_ORDERING_VALUE` is a better name since it indicates commit time ordering and it should be used only in that way. `DEFAULT` or `NON_EXISTANT` might not be obvious, and may lead to confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2609172573
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5f919ce39525f4ca5a7f94fce87a99b0caea430 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348)
* 0981d7e13ec076599c15edaf74eb46258f968304 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529234518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ec55f23e0ed3d9f17f4981276d818b0fda5723d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529467268
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ec55f23e0ed3d9f17f4981276d818b0fda5723d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340)
* deec65c355f2cc65234f00cfb86cec18cdc6de98 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529428159
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ec55f23e0ed3d9f17f4981276d818b0fda5723d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929633365
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -115,9 +114,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
Review Comment:
HUDI-8851 will be fixed by HUDI-8551 by handling precombine at the writer side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614542975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1927956967
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -115,9 +114,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
Review Comment:
are we setting the ordering field to empty for regular ingestion writes itself?
But how about backwards compatability?
if someone was using overwrite w/ latest payload in 0.15.0, but has ordering config set, we are forcing them to change to EVENT_TIME_ORDERING. but its a table property right?
can you help me understand
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
I feel "COMMIT_TIME_ORDERING_VALUE" is confusing to me. bcoz, when someone is using OverwriteWithLatestAvroPayload, we could still have valid ordering values. the value 0 is used, only when ordering value is missing from the record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1928095372
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -115,9 +114,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
Review Comment:
We discussed that if the table version is 6 (Hudi 0.14/0.15 release), from the `HoodieTableConfig`, it should serve `EVENT_TIME_ORDERING` as the merge mode if the payload class is `OverwriteWithLatestAvroPayload` with precombine field in the table config, so that there is no behavior change of reading the table of version 6 with Hudi 1.x in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2609436010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0981d7e13ec076599c15edaf74eb46258f968304 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929571227
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.DeleteRecord;
+
+import org.junit.jupiter.api.Test;
+
+import static org.apache.hudi.common.model.HoodieRecord.COMMIT_TIME_ORDERING_VALUE;
+import static org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer.getOrderingValue;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests {@link HoodieBaseFileGroupRecordBuffer}
+ */
+public class TestHoodieFileGroupRecordBuffer {
+ @Test
+ void testGetOrderingValueFromDeleteRecord() {
Review Comment:
does this test fails w/ your source code changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614170617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
}, {
"hash" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074",
"triggerID" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"triggerType" : "PUSH"
}, {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1883296568
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -226,26 +227,22 @@ public String getRecordKey(T record, Schema schema) {
* @param metadataMap A map containing the record metadata.
* @param schema The Avro schema of the record.
* @param orderingFieldName name of the ordering field
- * @param orderingFieldTypeOpt type of the ordering field
- * @param orderingFieldDefault default value for ordering
* @return The ordering value.
*/
public Comparable getOrderingValue(Option<T> recordOption,
Map<String, Object> metadataMap,
Schema schema,
- String orderingFieldName,
- Option<Schema.Type> orderingFieldTypeOpt,
- Comparable orderingFieldDefault) {
+ Option<String> orderingFieldName) {
if (metadataMap.containsKey(INTERNAL_META_ORDERING_FIELD)) {
return (Comparable) metadataMap.get(INTERNAL_META_ORDERING_FIELD);
}
- if (!recordOption.isPresent() || !orderingFieldTypeOpt.isPresent()) {
- return orderingFieldDefault;
+ if (!recordOption.isPresent() || orderingFieldName.isEmpty()) {
Review Comment:
whats the case where recordOption could be empty?
are there chances that ordering col is set in options and is of type String (non integer) and recordOption is empty? won't we return 0 (integer) as the default from here.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -116,9 +115,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
+ ? Option.empty()
+ : Option.ofNullable(ConfigUtils.getOrderingField(props))
+ .or(() -> {
+ String preCombineField = hoodieTableMetaClient.getTableConfig().getPreCombineField();
+ if (StringUtils.isNullOrEmpty(preCombineField)) {
Review Comment:
Option.ofNullable() could simplify L 123 to 126.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestMergeIntoTable.scala:
##########
@@ -543,7 +543,7 @@ class TestMergeIntoTable extends HoodieSparkSqlTestBase with ScalaAssertionSuppo
s"""
| merge into $tableName t0
| using (
- | select 2 as s_id, 'a2' as s_name, 15 as s_price, 1001 as ts, '2021-03-21' as dt
+ | select 2 as s_id, 'a2' as s_name, 15 as s_price, 1001L as ts, '2021-03-21' as dt
Review Comment:
why do we need this fixes (1001 -> 1001L) ?
can you throw some light please
##########
hudi-common/src/main/java/org/apache/hudi/common/model/EventTimeAvroPayload.java:
##########
@@ -41,7 +42,7 @@ public EventTimeAvroPayload(GenericRecord record, Comparable orderingVal) {
}
public EventTimeAvroPayload(Option<GenericRecord> record) {
- this(record.isPresent() ? record.get() : null, 0); // natural order
+ this(record.isPresent() ? record.get() : null, COMMIT_TIME_ORDERING_VALUE); // natural order
Review Comment:
so, in general we assume integer as default even if ordering col chosen is string or other data types?
and we never broke anything so far is it?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
why can't we name this as DEFAULT_ORDERING_VALUE or NON_EXISTANT_ORDERING_VALUE
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java:
##########
@@ -264,12 +270,27 @@ public UnaryOperator<ArrayWritable> projectRecord(Schema from, Schema to, Map<St
}
@Override
- public Comparable castValue(Comparable value, Schema.Type newType) {
- //TODO: [HUDI-8261] actually do casting here
+ public Comparable convertValueToEngineType(Comparable value) {
if (value instanceof WritableComparable) {
return value;
}
- return (WritableComparable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(value, Schema.create(newType));
+ //TODO: [HUDI-8261] cover more types
+ if (value == null) {
+ return null;
+ } else if (value instanceof String) {
+ return new Text((String) value);
+ } else if (value instanceof Integer) {
+ return new IntWritable((int) value);
+ } else if (value instanceof Long) {
+ return new LongWritable((long) value);
+ } else if (value instanceof Float) {
+ return new FloatWritable((float) value);
+ } else if (value instanceof Double) {
+ return new DoubleWritable((double) value);
+ } else if (value instanceof Boolean) {
Review Comment:
I don't even know if these datatypes will work for precombine only. since default value is picked as 0.
can we test it out and updates docs on known limitations if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2612123228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
}, {
"hash" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074",
"triggerID" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72698b8926bf4605f30c53c8c753fbfe55d6144c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929631217
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java:
##########
@@ -56,6 +56,7 @@ public abstract class HoodieRecord<T> implements HoodieRecordCompatibilityInterf
public static final String FILENAME_METADATA_FIELD = HoodieMetadataField.FILENAME_METADATA_FIELD.getFieldName();
public static final String OPERATION_METADATA_FIELD = HoodieMetadataField.OPERATION_METADATA_FIELD.getFieldName();
public static final String HOODIE_IS_DELETED_FIELD = "_hoodie_is_deleted";
+ public static final int COMMIT_TIME_ORDERING_VALUE = 0;
Review Comment:
After the discussion, this is renamed to `DEFAULT_ORDERING_VALUE` to foster better code readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929128715
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -115,9 +114,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
Review Comment:
btw, can we check if this fix will take care of https://issues.apache.org/jira/browse/HUDI-8851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614146024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
}, {
"hash" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074",
"triggerID" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"triggerType" : "PUSH"
}, {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72698b8926bf4605f30c53c8c753fbfe55d6144c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074)
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2529659924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* deec65c355f2cc65234f00cfb86cec18cdc6de98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926512752
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -226,26 +227,22 @@ public String getRecordKey(T record, Schema schema) {
* @param metadataMap A map containing the record metadata.
* @param schema The Avro schema of the record.
* @param orderingFieldName name of the ordering field
- * @param orderingFieldTypeOpt type of the ordering field
- * @param orderingFieldDefault default value for ordering
* @return The ordering value.
*/
public Comparable getOrderingValue(Option<T> recordOption,
Map<String, Object> metadataMap,
Schema schema,
- String orderingFieldName,
- Option<Schema.Type> orderingFieldTypeOpt,
- Comparable orderingFieldDefault) {
+ Option<String> orderingFieldName) {
if (metadataMap.containsKey(INTERNAL_META_ORDERING_FIELD)) {
return (Comparable) metadataMap.get(INTERNAL_META_ORDERING_FIELD);
}
- if (!recordOption.isPresent() || !orderingFieldTypeOpt.isPresent()) {
- return orderingFieldDefault;
+ if (!recordOption.isPresent() || orderingFieldName.isEmpty()) {
Review Comment:
Only if the record is a delete record, `recordOption` is empty. If that's the case, and the ordering field is not set, we should use commit time ordering for such deletes. If the delete record comes with an ordering value, the `metadataMap` contains the ordering value already, i.e., processed by
```
if (metadataMap.containsKey(INTERNAL_META_ORDERING_FIELD)) {
return (Comparable) metadataMap.get(INTERNAL_META_ORDERING_FIELD);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926514387
##########
hudi-common/src/main/java/org/apache/hudi/common/model/EventTimeAvroPayload.java:
##########
@@ -41,7 +42,7 @@ public EventTimeAvroPayload(GenericRecord record, Comparable orderingVal) {
}
public EventTimeAvroPayload(Option<GenericRecord> record) {
- this(record.isPresent() ? record.get() : null, 0); // natural order
+ this(record.isPresent() ? record.get() : null, COMMIT_TIME_ORDERING_VALUE); // natural order
Review Comment:
Yes, we have special handling on `(integer) 0` even if the ordering field type is different. So nothing is broken here. This PR keeps the same notion of `(integer) 0` as the commit-time ordering as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1926530696
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestMergeIntoTable.scala:
##########
@@ -543,7 +543,7 @@ class TestMergeIntoTable extends HoodieSparkSqlTestBase with ScalaAssertionSuppo
s"""
| merge into $tableName t0
| using (
- | select 2 as s_id, 'a2' as s_name, 15 as s_price, 1001 as ts, '2021-03-21' as dt
+ | select 2 as s_id, 'a2' as s_name, 15 as s_price, 1001L as ts, '2021-03-21' as dt
Review Comment:
The `ts` field is in `long` type so the input data cannot be integer. Previously we made it work by doing the type cast; however that masks the issue and the user should always make sure the field types of the input data are the same as the table schema when using Spark SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614146532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3045",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
}, {
"hash" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074",
"triggerID" : "72698b8926bf4605f30c53c8c753fbfe55d6144c",
"triggerType" : "PUSH"
}, {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72698b8926bf4605f30c53c8c753fbfe55d6144c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3074)
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929633512
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.DeleteRecord;
+
+import org.junit.jupiter.api.Test;
+
+import static org.apache.hudi.common.model.HoodieRecord.COMMIT_TIME_ORDERING_VALUE;
+import static org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer.getOrderingValue;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests {@link HoodieBaseFileGroupRecordBuffer}
+ */
+public class TestHoodieFileGroupRecordBuffer {
+ @Test
+ void testGetOrderingValueFromDeleteRecord() {
Review Comment:
There is no behavior change but rather API simplification so I added these tests to make sure the assumption made in the file group record buffer is not changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614513838
follow up on CI failures and you are good to land
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2609169313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
}, {
"hash" : "0981d7e13ec076599c15edaf74eb46258f968304",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0981d7e13ec076599c15edaf74eb46258f968304",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5f919ce39525f4ca5a7f94fce87a99b0caea430 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348)
* 0981d7e13ec076599c15edaf74eb46258f968304 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12452:
URL: https://github.com/apache/hudi/pull/12452#discussion_r1929618161
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java:
##########
@@ -115,9 +114,16 @@ public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
} else {
this.payloadClass = Option.empty();
}
- this.orderingFieldName = Option.ofNullable(ConfigUtils.getOrderingField(props)).orElseGet(() -> hoodieTableMetaClient.getTableConfig().getPreCombineField());
- this.orderingFieldTypeOpt = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING ? Option.empty() : AvroSchemaUtils.findNestedFieldType(readerSchema, this.orderingFieldName);
- this.orderingFieldDefault = orderingFieldTypeOpt.map(type -> readerContext.castValue(0, type)).orElse(0);
+ this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
Review Comment:
Also note that, even prior to this PR, we're not using the ordering field if the merge mode is `COMMIT_TIME_ORDERING`. The change here just to make the assumption tight and not cause any confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614564360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614542459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "209541fa82f326cee90bcbf55cd691ee90ab6b4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 209541fa82f326cee90bcbf55cd691ee90ab6b4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2614541949
CI failed due to flakiness. Retrying CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2530176367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5f919ce39525f4ca5a7f94fce87a99b0caea430 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8682] Simplify precombine and ordering field value handling in the file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12452:
URL: https://github.com/apache/hudi/pull/12452#issuecomment-2530077445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2340",
"triggerID" : "4ec55f23e0ed3d9f17f4981276d818b0fda5723d",
"triggerType" : "PUSH"
}, {
"hash" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2342",
"triggerID" : "deec65c355f2cc65234f00cfb86cec18cdc6de98",
"triggerType" : "PUSH"
}, {
"hash" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345",
"triggerID" : "10f082f6ed1b23f9357e0d5d833653516d5f1145",
"triggerType" : "PUSH"
}, {
"hash" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5f919ce39525f4ca5a7f94fce87a99b0caea430",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10f082f6ed1b23f9357e0d5d833653516d5f1145 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2345)
* e5f919ce39525f4ca5a7f94fce87a99b0caea430 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org