You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/09 07:50:15 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
nsivabalan opened a new pull request #2311:
URL: https://github.com/apache/hudi/pull/2311
## What is the purpose of the pull request
Existing OverwriteWithLatestAvroPayload always chooses incoming record when combineAndGetUpdateValue is called, but there are chances a record is late arriving and needs to be ignored. Hence adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue.
## Brief change log
- Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue.
- Added config "hoodie.datasource.honor.ordering.while.merging" which will determine to use OverwriteWithLatestAvroPayload or OverwriteWithLatestAvroPayloadV1.
## Verify this pull request
This change added tests and can be verified as follows:
- *Added tests tp TestDataSourceDefaults to verify the change.*
- *Added TestOverwriteWithLatestAvroPayloadV1 to0 verify the new payload class*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-741799295
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=h1) Report
> Merging [#2311](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=desc) (d932723) into [master](https://codecov.io/gh/apache/hudi/commit/26cdc457f6b2491caf28dcb7e33f03e7a2dac6fa?el=desc) (26cdc45) will **decrease** coverage by `42.30%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2311 +/- ##
=============================================
- Coverage 52.35% 10.04% -42.31%
+ Complexity 2631 48 -2583
=============================================
Files 329 52 -277
Lines 14808 1852 -12956
Branches 1484 223 -1261
=============================================
- Hits 7752 186 -7566
+ Misses 6441 1653 -4788
+ Partials 615 13 -602
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.04% <0.00%> (-60.08%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `0.00% <0.00%> (-70.55%)` | `0.00 <0.00> (-49.00)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [306 more](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r543973549
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
Review comment:
actually we can't do that. it could be null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-741600626
@vinothchandar : can you take a look at this patch. I am yet to find a good name for the new class added (OverwriteWithLatestAvroPayloadV1). Ignoring that for now, basically I have introduced a new config called "hoodie.datasource.honor.ordering.while.merging". Based on the value set, either existing OverwriteWithLatestAvroPayload will be used or the new class will be used while instantiating HoodiePayloads. By this way, callers of combineAndGetUpdateValue don't need to pass around the properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on a change in pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r539130578
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayloadV1.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field 2.
+ * combineAndGetUpdateValue/getInsertValue - Simply overwrites storage with latest delta record
+ */
+public class OverwriteWithLatestAvroPayloadV1 extends BaseAvroPayload
+ implements HoodieRecordPayload<OverwriteWithLatestAvroPayloadV1> {
+
+ private Map<String, String> props;
Review comment:
should we use a string to save storage? Looks like this will create a map for every record.
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -205,6 +205,12 @@ object DataSourceWriteOptions {
val PRECOMBINE_FIELD_OPT_KEY = "hoodie.datasource.write.precombine.field"
val DEFAULT_PRECOMBINE_FIELD_OPT_VAL = "ts"
+ /**
+ * Field when set to true, will honor ordering while merging two records. If not, incoming record will overwrite
+ * record in storage.
+ */
+ val HONOR_ORDERING_WHILE_MERGING_OPT_KEY = "hoodie.datasource.honor.ordering.while.merging"
+ val DEFAULT_HONOR_ORDERING_WHILE_MERGING_OPT_VAL = "false"
Review comment:
keep this `true` is more rational for me. From a new user perspective, I don't really understand the difference between `preCombine` and `combineWithLatestValue`. It took me a while to figure this out and originally I thought this is a bug. But we need to think about the impact of the existing user.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -121,6 +121,10 @@
private static final String MERGE_DATA_VALIDATION_CHECK_ENABLED = "hoodie.merge.data.validation.enabled";
private static final String DEFAULT_MERGE_DATA_VALIDATION_CHECK_ENABLED = "false";
+ // payload ordering field
+ private static final String PAYLOAD_ORDERING_FIELD_PROP = "hoodie.payload.ordering.field";
Review comment:
Is there any reason we need a different config from the pre-combine field? IIUC we wanna make the merging logic consistent, so `preCombine` and `combineAndGetUpdateValue` should produce the same result?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayloadV1.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field 2.
+ * combineAndGetUpdateValue/getInsertValue - Simply overwrites storage with latest delta record
+ */
+public class OverwriteWithLatestAvroPayloadV1 extends BaseAvroPayload
Review comment:
naming is hard... how about `CombineWithLargestOrderingValPayload`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-742839517
@vinothchandar : can you take a high level pass and let me know if this looks good. or do we need to have the map passed in as an arg to combineAndGetUpdateValue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r543467964
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePayloadProps.java
##########
@@ -0,0 +1,12 @@
+package org.apache.hudi.common.model;
+
+/**
+ * Since both payload classes and HoodiePayloadConfig needs to access these props, storing it here.
+ */
+public class HoodiePayloadProps {
+
+ // payload ordering field
Review comment:
more descriptive doc?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
+ Comparable incomingOrderingVal = (Comparable) getNestedFieldVal(incomingRecord, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), false);
+
+ // Null check is needed here to support schema evolution. The record in storage may be from old schema where
+ // the new ordering column might not be present and hence returns null.
+ if (persistedOrderingVal != null && ((Comparable) persistedOrderingVal).compareTo(incomingOrderingVal) > 0) {
Review comment:
nts: the check means persistedOrderingVal > incomingOrderingVal i.e we should retain the value on disk
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
Review comment:
I think we should just refer to the other method here in the javadoc, instead of repeating the entire description here again.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
Review comment:
close the `</p>`
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
Review comment:
just do the `Comparable` cast here itself?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
+ * and merge them with current image for a database row on storage
*
* @param currentValue Current value in storage, to merge/combine this payload with
* @param schema Schema used for record
* @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException;
/**
- * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a
- * new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e
- * insert) Return EMPTY to skip writing this record.
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object. This method takes in a property map as an arg so that implementation can decide their business logic based
+ * * on some properties set.
+ * <p>
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
+ * and merge them with current image for a database row on storage
+ *
+ * @param currentValue Current value in storage, to merge/combine this payload with
+ * @param schema Schema used for record
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
*/
+ default Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ return combineAndGetUpdateValue(currentValue, schema);
+ }
+
+ /**
+ * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given
+ * HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record.
+ * @param schema Schema used for record
+ * @return the {@link IndexedRecord} to be inserted.
+ */
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Option<IndexedRecord> getInsertValue(Schema schema) throws IOException;
/**
- * This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to
- * {@code WriteStatus.markSuccess()} and {@code WriteStatus.markFailure()} in order to compute some aggregate metrics
- * using the metadata in the context of a write success or failure.
+ * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given
+ * HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record.
+ * This method takes in a property map as an arg so that implementation can decide their business logic based on some properties set.
Review comment:
this line can be bit pithy?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
Review comment:
this is not just for delta streamer.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
Review comment:
is this class pretty much a modified version of `OverwriteWithLatestAvroPayload`? if so, can we reuse some code by having that extend from this, and override methods as needed?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
Review comment:
same here
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -729,6 +731,10 @@ public FileSystemViewStorageConfig getClientSpecifiedViewStorageConfig() {
return clientSpecifiedViewStorageConfig;
}
+ public HoodiePayloadConfig getHoodiePayloadConfig() {
Review comment:
rename: getPayloadConfig()
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BaseAvroPayload.java
##########
@@ -29,6 +29,7 @@
* Base class for all AVRO record based payloads, that can be ordered based on a field.
*/
public abstract class BaseAvroPayload implements Serializable {
+
Review comment:
nit: extra line
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
Review comment:
please remove "if combinng turned on ..." comment. we should ideally refrain from referring to higher level constructs from lower layers.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
+ * and merge them with current image for a database row on storage
*
* @param currentValue Current value in storage, to merge/combine this payload with
* @param schema Schema used for record
* @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Review comment:
mark all deprecated methods with the right ApiMaturityLevel.DEPRECATED?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
+ * and merge them with current image for a database row on storage
*
* @param currentValue Current value in storage, to merge/combine this payload with
* @param schema Schema used for record
* @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException;
/**
- * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a
- * new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e
- * insert) Return EMPTY to skip writing this record.
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object. This method takes in a property map as an arg so that implementation can decide their business logic based
+ * * on some properties set.
+ * <p>
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
+ * and merge them with current image for a database row on storage
+ *
+ * @param currentValue Current value in storage, to merge/combine this payload with
+ * @param schema Schema used for record
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return new combined/merged value to be written back to storage. EMPTY to skip writing this record.
*/
+ default Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ return combineAndGetUpdateValue(currentValue, schema);
+ }
+
+ /**
+ * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given
+ * HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record.
+ * @param schema Schema used for record
+ * @return the {@link IndexedRecord} to be inserted.
+ */
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Option<IndexedRecord> getInsertValue(Schema schema) throws IOException;
/**
- * This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to
- * {@code WriteStatus.markSuccess()} and {@code WriteStatus.markFailure()} in order to compute some aggregate metrics
- * using the metadata in the context of a write success or failure.
+ * Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given
+ * HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record.
+ * This method takes in a property map as an arg so that implementation can decide their business logic based on some properties set.
+ * @param schema Schema used for record
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the {@link IndexedRecord} to be inserted.
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default Option<IndexedRecord> getInsertValue(Schema schema, Properties properties) throws IOException {
+ return getInsertValue(schema);
+ }
+
+ /**
+ * This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to {@code WriteStatus.markSuccess()} and
Review comment:
thanks for fixing this
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java
##########
@@ -29,47 +29,93 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import java.util.Properties;
/**
- * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which
- * depend on record specific logic.
+ * Every Hoodie table has an implementation of the <code>HoodieRecordPayload</code> This abstracts out callbacks which depend on record specific logic.
*/
@PublicAPIClass(maturity = ApiMaturityLevel.STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload> extends Serializable {
/**
- * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to
- * insert/upsert (if combining turned on in HoodieClientConfig).
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig).
*/
+ @Deprecated
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
T preCombine(T another);
/**
- * This methods lets you write custom merging/combining logic to produce new values as a function of current value on
- * storage and whats contained in this object.
+ * When more than one HoodieRecord have the same HoodieKey, this function combines them before attempting to insert/upsert (if combining turned on
+ * in HoodieClientConfig) by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.
+ * @param another instance of another {@link HoodieRecordPayload} to be combined with.
+ * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+ * @return the combined value
+ */
+ @PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
+ default T preCombine(T another, Properties properties) {
+ return preCombine(another);
+ }
+
+ /**
+ * This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained
+ * in this object.
* <p>
- * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You
- * may be reading DB redo logs, and merge them with current image for a database row on storage
+ * eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs,
Review comment:
place 2 on a newline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r543972882
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
Review comment:
I thought you had already reviewed previous PR and hence just copied as is. Will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-745797881
yes, you are right. we are **NOT** making this new payload default for now. Have created https://issues.apache.org/jira/browse/HUDI-1464 to track this.
btw, have addressed all feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r543976521
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
+ Comparable incomingOrderingVal = (Comparable) getNestedFieldVal(incomingRecord, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), false);
+
+ // Null check is needed here to support schema evolution. The record in storage may be from old schema where
+ // the new ordering column might not be present and hence returns null.
+ if (persistedOrderingVal != null && ((Comparable) persistedOrderingVal).compareTo(incomingOrderingVal) > 0) {
Review comment:
yes . nts = note to self. sorry :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r540541958
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayloadV1.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field 2.
+ * combineAndGetUpdateValue/getInsertValue - Simply overwrites storage with latest delta record
+ */
+public class OverwriteWithLatestAvroPayloadV1 extends BaseAvroPayload
+ implements HoodieRecordPayload<OverwriteWithLatestAvroPayloadV1> {
+
+ private Map<String, String> props;
Review comment:
yeah. lets wait to hear from @vinothchandar . Previous impl was sending in this map as arg to combineAndGetUpdateValue rather than storing it as instance var. may be thats better in terms of storage cost. I thought we could avoid making changes to all callers of combineAndGetUpdateValue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-743855911
@vinothchandar @garyli1019 : I did some revamp based on feedback. Patch is ready to be reviewed again.
Open to naming the new class differently. Also, will fix java docs for new class based on it. As of now, it says default payload class used in deltastreamer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r540771309
##########
File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
##########
@@ -142,6 +142,20 @@ public static HoodieRecordPayload createPayload(String payloadClass, GenericReco
}
}
+ /**
+ * Create a payload class via reflection, passing in an ordering/precombine value.
+ */
+ public static HoodieRecordPayload createPayload(String payloadClass, GenericRecord record, Comparable orderingVal,
Review comment:
@nsivabalan won't this break for an existing payloadClass (user defined), that does have this three member constructor?
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -154,9 +154,16 @@ private[hudi] object HoodieSparkSqlWriter {
val hoodieRecord = if (shouldCombine) {
val orderingVal = HoodieAvroUtils.getNestedFieldVal(gr, parameters(PRECOMBINE_FIELD_OPT_KEY), false)
.asInstanceOf[Comparable[_]]
- DataSourceUtils.createHoodieRecord(gr,
- orderingVal, keyGenerator.getKey(gr),
- parameters(PAYLOAD_CLASS_OPT_KEY))
+ val honorOrderingWhileMerging = parameters(HONOR_ORDERING_WHILE_MERGING_OPT_KEY).toBoolean
Review comment:
this is more and more overhead for the user to learn one more parameter when using one specific payload
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -205,6 +205,12 @@ object DataSourceWriteOptions {
val PRECOMBINE_FIELD_OPT_KEY = "hoodie.datasource.write.precombine.field"
val DEFAULT_PRECOMBINE_FIELD_OPT_VAL = "ts"
+ /**
+ * Field when set to true, will honor ordering while merging two records. If not, incoming record will overwrite
+ * record in storage.
+ */
+ val HONOR_ORDERING_WHILE_MERGING_OPT_KEY = "hoodie.datasource.honor.ordering.while.merging"
+ val DEFAULT_HONOR_ORDERING_WHILE_MERGING_OPT_VAL = "false"
Review comment:
I really don't like having these payload specific configs, top level in datasource options.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-741600626
@vinothchandar : can you take a look at this patch. I am yet to find a good name for the class. Ignoring that for now, basically I have introduced a new config called "hoodie.datasource.honor.ordering.while.merging". Based on the value set, either existing OverwriteWithLatestAvroPayload will be used or the new class will be used while instantiating HoodiePayloads. By this way, callers of combineAndGetUpdateValue don't need to pass around the properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-747703536
@nsivabalan could you check the build?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r543974062
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field
+ * 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.
+ */
+public class DefaultHoodieRecordPayload extends BaseAvroPayload
+ implements HoodieRecordPayload<DefaultHoodieRecordPayload> {
+
+ public DefaultHoodieRecordPayload(GenericRecord record, Comparable orderingVal) {
+ super(record, orderingVal);
+ }
+
+ public DefaultHoodieRecordPayload(Option<GenericRecord> record) {
+ this(record.isPresent() ? record.get() : null, (record1) -> 0); // natural order
+ }
+
+ @Override
+ public DefaultHoodieRecordPayload preCombine(DefaultHoodieRecordPayload another) {
+ // pick the payload with greatest ordering value
+ if (another.orderingVal.compareTo(orderingVal) > 0) {
+ return another;
+ } else {
+ return this;
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ IndexedRecord indexedRecord = bytesToAvro(recordBytes, schema);
+ if (isDeleteRecord((GenericRecord) indexedRecord)) {
+ return Option.empty();
+ } else {
+ return Option.of(indexedRecord);
+ }
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) throws IOException{
+ return getInsertValue(schema);
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
+ if (recordBytes.length == 0) {
+ return Option.empty();
+ }
+ GenericRecord incomingRecord = bytesToAvro(recordBytes, schema);
+ /*
+ * Combining strategy here returns currentValue on disk if incoming record is older.
+ * The incoming record can be either a delete (sent as an upsert with _hoodie_is_deleted set to true)
+ * or an insert/update record. In any case, if it is older than the record in disk, the currentValue
+ * in disk is returned (to be rewritten with new commit time).
+ *
+ * NOTE: Deletes sent via EmptyHoodieRecordPayload and/or Delete operation type do not hit this code path
+ * and need to be dealt with separately.
+ */
+ Object persistedOrderingVal = getNestedFieldVal((GenericRecord) currentValue, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), true);
+ Comparable incomingOrderingVal = (Comparable) getNestedFieldVal(incomingRecord, properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP), false);
+
+ // Null check is needed here to support schema evolution. The record in storage may be from old schema where
+ // the new ordering column might not be present and hence returns null.
+ if (persistedOrderingVal != null && ((Comparable) persistedOrderingVal).compareTo(incomingOrderingVal) > 0) {
Review comment:
sorry, you wanted to confirm this behavior? can you please clarify.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#discussion_r540542497
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayloadV1.java
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import static org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro;
+import static org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+
+/**
+ * Default payload used for delta streamer.
+ * <p>
+ * 1. preCombine - Picks the latest delta record for a key, based on an ordering field 2.
+ * combineAndGetUpdateValue/getInsertValue - Simply overwrites storage with latest delta record
+ */
+public class OverwriteWithLatestAvroPayloadV1 extends BaseAvroPayload
Review comment:
yeah. Ideally existing one should have been OverwriteWithIncomingAvroPayload and this new one should be OverwriteWithLatestAvroPayload.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2311: [HUDI-115] Adding OverwriteWithLatestAvroPayloadV1 to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-741799295
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=h1) Report
> Merging [#2311](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=desc) (d637a72) into [master](https://codecov.io/gh/apache/hudi/commit/319b7a58e4773ca51c4ef9b82faf55d03d208e46?el=desc) (319b7a5) will **decrease** coverage by `43.16%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2311 +/- ##
=============================================
- Coverage 53.52% 10.35% -43.17%
+ Complexity 2789 48 -2741
=============================================
Files 355 51 -304
Lines 16169 1786 -14383
Branches 1650 213 -1437
=============================================
- Hits 8654 185 -8469
+ Misses 6815 1588 -5227
+ Partials 700 13 -687
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudispark | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `10.35% <ø> (-59.75%)` | `0.00 <ø> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2311?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| ... and [326 more](https://codecov.io/gh/apache/hudi/pull/2311/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2311:
URL: https://github.com/apache/hudi/pull/2311#issuecomment-741799295
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #2311: [HUDI-115] Adding DefaultHoodieRecordPayload to honor ordering with combineAndGetUpdateValue
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2311:
URL: https://github.com/apache/hudi/pull/2311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org