You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2024/11/26 21:08:31 UTC
[PR] [HUDI-123456] add schema handler tests and refactor fg reader a bit for more clarity [hudi]
jonvex opened a new pull request, #12340:
URL: https://github.com/apache/hudi/pull/12340
### Change Logs
safdsafd
### Impact
sadfsdagfa
### Risk level (write none, low medium or high below)
none
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3078257960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* e420f4735b6fc921050e4e2805c90b2479e937e5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631)
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-123456] add schema handler tests and refactor fg reader a bit for more clarity [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2501941276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f7b5f7f2b6485fbac119f644699292a1009ed2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2520987474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c03776da2feacc0426c98079fc6393dc0a6838ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2770219684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2027322690
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
Review Comment:
That's what testCow is. We don't actually make any distinction on what table type it is. We just need to know if it has log files or has bootstrap files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776270951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3046129328
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 4dd4ea2a3b85d60ff48850130a9336b9df5d244c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604)
* e420f4735b6fc921050e4e2805c90b2479e937e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2192771404
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
Review Comment:
I see a handful of places in this test that assert on `readerContext.getNeedsBootstrapMerge()`, is it updating throughout the test case? I did not think so on first read so want to make sure I'm understanding the test correctly
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupReaderSchemaHandler.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestFileGroupReaderSchemaHandler extends SchemaHandlerTestBase {
+
+ @Test
+ public void testCow() {
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
Review Comment:
Update usage of `Boolean.TRUE` to simply `true` now that the `populateMetaFields` returns a primitive
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static void setupMORTable(RecordMergeMode mergeMode, boolean hasPrecombine, HoodieTableConfig hoodieTableConfig) {
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ when(hoodieTableConfig.getTableVersion()).thenReturn(HoodieTableVersion.EIGHT);
+ }
+ }
+
+ private static HoodieRecordMerger mockRecordMerger(boolean isProjectionCompatible, String[] mandatoryFields) throws IOException {
+ HoodieRecordMerger merger = mock(HoodieRecordMerger.class);
+ when(merger.isProjectionCompatible()).thenReturn(isProjectionCompatible);
+ when(merger.merge(any(), any(), any(), any(), any())).thenReturn(null);
+ when(merger.getMandatoryFieldsForMerging(any(), any(), any())).thenReturn(mandatoryFields);
+ return merger;
+ }
+
+ static HoodieReaderContext<String> createReaderContext(HoodieTableConfig hoodieTableConfig, boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition, HoodieRecordMerger merger) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(hoodieTableConfig, supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ readerContext.setRecordMerger(Option.ofNullable(merger));
+ return readerContext;
+ }
+
+ static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
Review Comment:
nitpick, but we can make this typed as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776209373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2210169817
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static void setupMORTable(RecordMergeMode mergeMode, boolean hasPrecombine, HoodieTableConfig hoodieTableConfig) {
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ when(hoodieTableConfig.getTableVersion()).thenReturn(HoodieTableVersion.EIGHT);
+ }
+ }
+
+ private static HoodieRecordMerger mockRecordMerger(boolean isProjectionCompatible, String[] mandatoryFields) throws IOException {
+ HoodieRecordMerger merger = mock(HoodieRecordMerger.class);
+ when(merger.isProjectionCompatible()).thenReturn(isProjectionCompatible);
+ when(merger.merge(any(), any(), any(), any(), any())).thenReturn(null);
+ when(merger.getMandatoryFieldsForMerging(any(), any(), any())).thenReturn(mandatoryFields);
+ return merger;
+ }
+
+ static HoodieReaderContext<String> createReaderContext(HoodieTableConfig hoodieTableConfig, boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition, HoodieRecordMerger merger) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(hoodieTableConfig, supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ readerContext.setRecordMerger(Option.ofNullable(merger));
+ return readerContext;
+ }
+
+ static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
Review Comment:
Addressed
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupReaderSchemaHandler.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestFileGroupReaderSchemaHandler extends SchemaHandlerTestBase {
+
+ @Test
+ public void testCow() {
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
Review Comment:
Addressed
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static void setupMORTable(RecordMergeMode mergeMode, boolean hasPrecombine, HoodieTableConfig hoodieTableConfig) {
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ when(hoodieTableConfig.getTableVersion()).thenReturn(HoodieTableVersion.EIGHT);
+ }
+ }
+
+ private static HoodieRecordMerger mockRecordMerger(boolean isProjectionCompatible, String[] mandatoryFields) throws IOException {
+ HoodieRecordMerger merger = mock(HoodieRecordMerger.class);
+ when(merger.isProjectionCompatible()).thenReturn(isProjectionCompatible);
+ when(merger.merge(any(), any(), any(), any(), any())).thenReturn(null);
+ when(merger.getMandatoryFieldsForMerging(any(), any(), any())).thenReturn(mandatoryFields);
+ return merger;
+ }
+
+ static HoodieReaderContext<String> createReaderContext(HoodieTableConfig hoodieTableConfig, boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition, HoodieRecordMerger merger) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(hoodieTableConfig, supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ readerContext.setRecordMerger(Option.ofNullable(merger));
+ return readerContext;
+ }
+
+ static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockReaderContext extends HoodieReaderContext<String> {
Review Comment:
Addressed. It will not be possible to mock since the class is modified by the handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3078751218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* 0306dba3da80fa63984356185c93d6e6ba9e9b54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3024009559
Ran the test `TestPositionBasedFileGroupRecordBuffer` locally 50 times. Could not find the failure reported by github CI. Azure CI had passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2779255966
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 95e3598fd2de8830b945234ee25ba328376068af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618)
* 91f7d3c60651beea6b7a99c82132e4d558c20f05 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #12340:
URL: https://github.com/apache/hudi/pull/12340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3026745293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2502047684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f7b5f7f2b6485fbac119f644699292a1009ed2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2520804800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f7b5f7f2b6485fbac119f644699292a1009ed2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983)
* c03776da2feacc0426c98079fc6393dc0a6838ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2520800845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f7b5f7f2b6485fbac119f644699292a1009ed2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983)
* c03776da2feacc0426c98079fc6393dc0a6838ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2769924056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e98e43dd1d1b5fa02c48f0eb91cda80256d7827 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558)
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2027325214
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
+ readerContext.setHasBootstrapBaseFile(false);
+ //has no effect on schema unless we support position based merging
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = supportsParquetRowIndex
+ ? new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? HoodiePositionBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = supportsParquetRowIndex
+ ? new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = DATA_SCHEMA;
Review Comment:
```
protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3022754211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 757813fe29f7c10735bfb07d6e35c2f0a624e792 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3026609024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 0e6cd937c9c0775fc29ab1c0ae15a265747891b3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564)
* 11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3089353122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
}, {
"hash" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* 0306dba3da80fa63984356185c93d6e6ba9e9b54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782)
* b7edb4cec061a4092427ebde53cef9cb899c1aa8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2001378215
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -165,7 +165,7 @@ class SparkFileFormatInternalRowReaderContext(parquetFileReader: SparkParquetRea
HoodieAvroUtils.removeFields(skeletonRequiredSchema, rowIndexColumn))
//If we need to do position based merging with log files we will leave the row index column at the end
- val dataProjection = if (getHasLogFiles && getShouldMergeUseRecordPosition) {
+ val dataProjection = if (getShouldMergeUseRecordPosition) {
Review Comment:
why removed the log files check?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
Review Comment:
I see lot of code duplications across. atleast all 3 COW related tests, we can dedup.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
Review Comment:
can we add a test for hasLogFiles = false. i.e. representing a compacted file group.
we should not be projecting the additional cols that we usually do when we have log files.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedSchemaHandler.java:
##########
@@ -50,10 +51,18 @@ public HoodiePositionBasedSchemaHandler(HoodieReaderContext<T> readerContext,
super(readerContext, dataSchema, requestedSchema, internalSchemaOpt, hoodieTableConfig, properties);
}
+ private boolean morMergeNeedsPositionCol() {
+ return readerContext.supportsParquetRowIndex() && readerContext.getShouldMergeUseRecordPosition();
+ }
+
+ private boolean bootstrapMergeNeedsPositionCol() {
Review Comment:
I don't have good context on bootstrap flows. Will let @yihua review/skim them
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedSchemaHandler.java:
##########
@@ -50,10 +51,18 @@ public HoodiePositionBasedSchemaHandler(HoodieReaderContext<T> readerContext,
super(readerContext, dataSchema, requestedSchema, internalSchemaOpt, hoodieTableConfig, properties);
}
+ private boolean morMergeNeedsPositionCol() {
Review Comment:
infact L 56 we are checking for log files prior to this patch. and w/ this patch we are calling morMergeNeedsPositionCol which is not checking for log files.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
Review Comment:
recently we landed a fix to account for "_hoodie_is_deleted" with https://github.com/apache/hudi/pull/12970
may be we can consolidate those tests here as well.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedSchemaHandler.java:
##########
@@ -50,10 +51,18 @@ public HoodiePositionBasedSchemaHandler(HoodieReaderContext<T> readerContext,
super(readerContext, dataSchema, requestedSchema, internalSchemaOpt, hoodieTableConfig, properties);
}
+ private boolean morMergeNeedsPositionCol() {
Review Comment:
why naming this as MOR? the two conditions we check are not strictly confined to MOR tables right?
```
readerContext.supportsParquetRowIndex() && readerContext.getShouldMergeUseRecordPosition()
```
or if we can include hasLogFiles, may be it might make sense.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
+ readerContext.setHasBootstrapBaseFile(false);
+ //has no effect on schema unless we support position based merging
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
Review Comment:
minor. can we add java docs to HoodieRecordMerger.isProjectionCompatible
that its only applicable for CUSTOM Merger. for commit itme ordering and event time ordering, this api is not used only. It was not apparent.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.HoodiePositionBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA,
+ requestedSchema, Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(false);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(true);
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(true);
+ readerContext.setShouldMergeUseRecordPosition(false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ HoodieFileGroupReaderSchemaHandler schemaHandler =
+ new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+
+ schemaHandler = new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex));
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(true);
+ readerContext.setHasBootstrapBaseFile(false);
+ //has no effect on schema unless we support position based merging
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = DATA_SCHEMA;
+ HoodieFileGroupReaderSchemaHandler schemaHandler = supportsParquetRowIndex
+ ? new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? HoodiePositionBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = supportsParquetRowIndex
+ ? new HoodiePositionBasedSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new HoodieFileGroupReaderSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = DATA_SCHEMA;
Review Comment:
don't we need to add meta fields at the beginning in addition to actual data cols?
for CUSTOM merge mode and if projection is not compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2047737400
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
Review Comment:
Let's make test classes for each schema handler implementation instead of bundling them together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3089727813
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
}, {
"hash" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6832",
"triggerID" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* b7edb4cec061a4092427ebde53cef9cb899c1aa8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6832)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2210172011
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
Review Comment:
Schema handler updates it here.
```
protected Schema prepareRequiredSchema() {
Schema preReorderRequiredSchema = generateRequiredSchema();
Pair<List<Schema.Field>, List<Schema.Field>> requiredFields = getDataAndMetaCols(preReorderRequiredSchema);
readerContext.setNeedsBootstrapMerge(readerContext.getHasBootstrapBaseFile()
&& !requiredFields.getLeft().isEmpty() && !requiredFields.getRight().isEmpty());
return readerContext.getNeedsBootstrapMerge()
? createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream()).collect(Collectors.toList()))
: preReorderRequiredSchema;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3046420074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* e420f4735b6fc921050e4e2805c90b2479e937e5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3047924556
@the-other-tim-brown Can you please take another look after the recent changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3026593556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 0e6cd937c9c0775fc29ab1c0ae15a265747891b3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564)
* 11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-123456] add schema handler tests and refactor fg reader a bit for more clarity [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2501939102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f7b5f7f2b6485fbac119f644699292a1009ed2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776153056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565)
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2560201871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c03776da2feacc0426c98079fc6393dc0a6838ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303)
* 8e98e43dd1d1b5fa02c48f0eb91cda80256d7827 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2560203389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c03776da2feacc0426c98079fc6393dc0a6838ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303)
* 8e98e43dd1d1b5fa02c48f0eb91cda80256d7827 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3024203644
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 757813fe29f7c10735bfb07d6e35c2f0a624e792 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550)
* 0e6cd937c9c0775fc29ab1c0ae15a265747891b3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3031596561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 4dd4ea2a3b85d60ff48850130a9336b9df5d244c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2192984012
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
Review Comment:
Where does the schema handler update in the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3046265687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 4dd4ea2a3b85d60ff48850130a9336b9df5d244c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604)
* e420f4735b6fc921050e4e2805c90b2479e937e5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3078354786
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* e420f4735b6fc921050e4e2805c90b2479e937e5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631)
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* 0306dba3da80fa63984356185c93d6e6ba9e9b54 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2180876048
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupReaderSchemaHandler.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestFileGroupReaderSchemaHandler extends SchemaHandlerTestBase {
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = mockReaderContext(false, false, false, false, null);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ when(readerContext.getNeedsBootstrapMerge()).thenReturn(false);
Review Comment:
We want to assert that readerContext.getNeedsBootstrapMerge() returns true because the schema handler should set that var. But using "when" at the end of the test doesn't do anything right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3022100495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 9036274b26750e16760b098f6894a42d2ef5f449 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535)
* 757813fe29f7c10735bfb07d6e35c2f0a624e792 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3019861906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 91f7d3c60651beea6b7a99c82132e4d558c20f05 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660)
* 646f07497bc45fb179cf3c2a7b979f54c13c4239 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2175469481
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
Review Comment:
Addressed
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2192968960
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
Review Comment:
Yes, that is set inside the schema handler. It is true if we are reading from both the base file and skeleton file. Hence we need to read those two files and concat/merge them together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776327099
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 95e3598fd2de8830b945234ee25ba328376068af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776148474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565)
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776550333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 95e3598fd2de8830b945234ee25ba328376068af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3089510663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
}, {
"hash" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6832",
"triggerID" : "b7edb4cec061a4092427ebde53cef9cb899c1aa8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* 0306dba3da80fa63984356185c93d6e6ba9e9b54 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6782)
* b7edb4cec061a4092427ebde53cef9cb899c1aa8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6832)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2779392747
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 91f7d3c60651beea6b7a99c82132e4d558c20f05 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2047860607
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
Review Comment:
You'll want to mock the table version for the config as well. Can you also extract the common mock setup for this test and `testMorBootstrap` to resusable methods?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
Review Comment:
You'll want to mock the table version for the config as well. Can you also extract the common mock setup for this test and `testMorBootstrap` to reusable methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2182128567
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupReaderSchemaHandler.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestFileGroupReaderSchemaHandler extends SchemaHandlerTestBase {
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = mockReaderContext(false, false, false, false, null);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ when(readerContext.getNeedsBootstrapMerge()).thenReturn(false);
Review Comment:
I had to do this to address Tim's comment on mocking the reader context. Did not know the bootstrap merge is set in the handler. I have reverted the mock related changes for reader context. There are test cases failing here when merge mode is custom and isProjectionCompatible is true for testMorBootstrap. Can you please help in debugging those? I am not sure why it is expected for bootstrap merge to be false in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776305063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 95e3598fd2de8830b945234ee25ba328376068af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3021662663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 646f07497bc45fb179cf3c2a7b979f54c13c4239 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532)
* 9036274b26750e16760b098f6894a42d2ef5f449 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3022155350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 9036274b26750e16760b098f6894a42d2ef5f449 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535)
* 757813fe29f7c10735bfb07d6e35c2f0a624e792 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3024456085
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 0e6cd937c9c0775fc29ab1c0ae15a265747891b3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2175469139
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, true, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static HoodieReaderContext<String> createReaderContext(boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ return readerContext;
+ }
+
+ private static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ private static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ private Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockMerger implements HoodieRecordMerger {
Review Comment:
Addressed
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, true, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static HoodieReaderContext<String> createReaderContext(boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ return readerContext;
+ }
+
+ private static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ private static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ private Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockMerger implements HoodieRecordMerger {
+
+ private final boolean isProjectionCompatible;
+ private final String[] mandatoryMergeFields;
+
+ MockMerger(boolean isProjectionCompatible, String[] mandatoryMergeFields) {
+ this.isProjectionCompatible = isProjectionCompatible;
+ this.mandatoryMergeFields = mandatoryMergeFields;
+ }
+
+ @Override
+ public boolean isProjectionCompatible() {
+ return this.isProjectionCompatible;
+ }
+
+ @Override
+ public String[] getMandatoryFieldsForMerging(Schema dataSchema, HoodieTableConfig cfg, TypedProperties properties) {
+ return this.mandatoryMergeFields;
+ }
+
+ @Override
+ public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer,
+ Schema newSchema, TypedProperties props) throws IOException {
+ return null;
+ }
+
+ @Override
+ public HoodieRecord.HoodieRecordType getRecordType() {
+ return null;
+ }
+
+ @Override
+ public String getMergingStrategy() {
+ return "";
+ }
+ }
+
+ static class MockReaderContext extends HoodieReaderContext<String> {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2047733315
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, true, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static HoodieReaderContext<String> createReaderContext(boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ return readerContext;
+ }
+
+ private static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ private static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ private Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockMerger implements HoodieRecordMerger {
Review Comment:
why can't you use a mock here?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestSchemaHandler.java:
##########
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.getPositionalMergeField;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestSchemaHandler {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ @Test
+ public void testCow() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, false, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = DATA_SCHEMA;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+ assertEquals(requestedSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrap() {
+ HoodieReaderContext<String> readerContext = createReaderContext(false, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, false);
+
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Collections.singletonList(getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @Test
+ public void testCowBootstrapWithPositionMerge() {
+ HoodieReaderContext<String> readerContext = createReaderContext(true, false, true, false);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ Schema requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, requestedSchema, hoodieTableConfig, true);
+ //meta cols must go first in the required schema
+ Schema expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_record_key"), getPositionalMergeField()), bootstrapFields.getLeft());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider"), getPositionalMergeField()), bootstrapFields.getRight());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, DATA_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(DATA_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertTrue(bootstrapFields.getLeft().isEmpty());
+ assertEquals(Arrays.asList(getField("begin_lat"), getField("tip_history"), getField("rider")), bootstrapFields.getRight());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ schemaHandler = createSchemaHandler(readerContext, DATA_SCHEMA, META_COLS_ONLY_SCHEMA, hoodieTableConfig, true);
+ assertEquals(META_COLS_ONLY_SCHEMA, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ assertEquals(Arrays.asList(getField("_hoodie_commit_seqno"), getField("_hoodie_record_key")), bootstrapFields.getLeft());
+ assertTrue(bootstrapFields.getRight().isEmpty());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ private static Stream<Arguments> testMorParams() {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean supportsParquetRowIndex : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ }
+ return b.build();
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, false, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ @ParameterizedTest
+ @MethodSource("testMorParams")
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieReaderContext<String> readerContext = createReaderContext(supportsParquetRowIndex, true, true, mergeUseRecordPosition);
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ Option<HoodieRecordMerger> merger = Option.of(new MockMerger(isProjectionCompatible, new String[]{"begin_lat", "begin_lon", "timestamp"}));
+ readerContext.setRecordMerger(merger);
+ }
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static HoodieReaderContext<String> createReaderContext(boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ return readerContext;
+ }
+
+ private static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ private static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ private Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockMerger implements HoodieRecordMerger {
+
+ private final boolean isProjectionCompatible;
+ private final String[] mandatoryMergeFields;
+
+ MockMerger(boolean isProjectionCompatible, String[] mandatoryMergeFields) {
+ this.isProjectionCompatible = isProjectionCompatible;
+ this.mandatoryMergeFields = mandatoryMergeFields;
+ }
+
+ @Override
+ public boolean isProjectionCompatible() {
+ return this.isProjectionCompatible;
+ }
+
+ @Override
+ public String[] getMandatoryFieldsForMerging(Schema dataSchema, HoodieTableConfig cfg, TypedProperties properties) {
+ return this.mandatoryMergeFields;
+ }
+
+ @Override
+ public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer,
+ Schema newSchema, TypedProperties props) throws IOException {
+ return null;
+ }
+
+ @Override
+ public HoodieRecord.HoodieRecordType getRecordType() {
+ return null;
+ }
+
+ @Override
+ public String getMergingStrategy() {
+ return "";
+ }
+ }
+
+ static class MockReaderContext extends HoodieReaderContext<String> {
Review Comment:
Same question here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2192746338
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.junit.jupiter.params.provider.Arguments;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.config.RecordMergeMode.COMMIT_TIME_ORDERING;
+import static org.apache.hudi.common.config.RecordMergeMode.CUSTOM;
+import static org.apache.hudi.common.config.RecordMergeMode.EVENT_TIME_ORDERING;
+import static org.apache.hudi.common.table.read.ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public abstract class SchemaHandlerTestBase {
+
+ protected static final Schema DATA_SCHEMA = HoodieAvroUtils.addMetadataFields(HoodieTestDataGenerator.AVRO_SCHEMA);
+ protected static final Schema DATA_SCHEMA_NO_DELETE = generateProjectionSchema(DATA_SCHEMA.getFields().stream()
+ .map(Schema.Field::name).filter(f -> !f.equals("_hoodie_is_deleted")).toArray(String[]::new));
+ protected static final Schema DATA_COLS_ONLY_SCHEMA = generateProjectionSchema("begin_lat", "tip_history", "rider");
+ protected static final Schema META_COLS_ONLY_SCHEMA = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key");
+
+ static Stream<Arguments> testMorParams(boolean supportsParquetRowIndex) {
+ Stream.Builder<Arguments> b = Stream.builder();
+ for (boolean mergeUseRecordPosition : new boolean[] {true, false}) {
+ for (boolean hasBuiltInDelete : new boolean[] {true, false}) {
+ b.add(Arguments.of(EVENT_TIME_ORDERING, true, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(EVENT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(COMMIT_TIME_ORDERING, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, true, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ b.add(Arguments.of(CUSTOM, false, false, mergeUseRecordPosition, supportsParquetRowIndex, hasBuiltInDelete));
+ }
+ }
+ return b.build();
+ }
+
+ public void testMor(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible,
+ isProjectionCompatible ? new String[] {"begin_lat", "begin_lon", "_hoodie_record_key", "timestamp"} : new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, false, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+
+ //read subset of columns
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "begin_lat", "tip_history", "rider", "_hoodie_record_key");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "_hoodie_record_key", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ }
+
+ public void testMorBootstrap(RecordMergeMode mergeMode,
+ boolean hasPrecombine,
+ boolean isProjectionCompatible,
+ boolean mergeUseRecordPosition,
+ boolean supportsParquetRowIndex,
+ boolean hasBuiltInDelete) throws IOException {
+ Schema dataSchema = hasBuiltInDelete ? DATA_SCHEMA : DATA_SCHEMA_NO_DELETE;
+ HoodieTableConfig hoodieTableConfig = mock(HoodieTableConfig.class);
+ setupMORTable(mergeMode, hasPrecombine, hoodieTableConfig);
+ HoodieRecordMerger merger = mockRecordMerger(isProjectionCompatible, new String[] {"begin_lat", "begin_lon", "timestamp"});
+ HoodieReaderContext<String> readerContext = createReaderContext(hoodieTableConfig, supportsParquetRowIndex, true, true, mergeUseRecordPosition, merger);
+ readerContext.setRecordMerger(Option.of(merger));
+ Schema requestedSchema = dataSchema;
+ FileGroupReaderSchemaHandler schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredFullSchema = supportsParquetRowIndex && mergeUseRecordPosition
+ ? ParquetRowIndexBasedSchemaHandler.addPositionalMergeCol(requestedSchema)
+ : requestedSchema;
+ assertEquals(expectedRequiredFullSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ Pair<List<Schema.Field>, List<Schema.Field>> bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ Pair<List<Schema.Field>, List<Schema.Field>> expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredFullSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ //read subset of columns
+ requestedSchema = generateProjectionSchema("begin_lat", "tip_history", "_hoodie_record_key", "rider");
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ Schema expectedRequiredSchema;
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_record_key", "begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ } else {
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only data cols
+ requestedSchema = DATA_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_record_key", "begin_lat", "tip_history", "rider");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("begin_lat", "tip_history", "rider", "begin_lon", "timestamp");
+ assertFalse(readerContext.getNeedsBootstrapMerge());
+ } else {
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ expectedRequiredSchema = dataSchema;
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (mergeMode == CUSTOM && isProjectionCompatible) {
+ if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ } else {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+
+ // request only meta cols
+ requestedSchema = META_COLS_ONLY_SCHEMA;
+ schemaHandler = createSchemaHandler(readerContext, dataSchema, requestedSchema, hoodieTableConfig, supportsParquetRowIndex);
+ if (mergeMode == EVENT_TIME_ORDERING && hasPrecombine) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == EVENT_TIME_ORDERING || mergeMode == COMMIT_TIME_ORDERING) {
+ expectedRequiredSchema = generateProjectionSchema(hasBuiltInDelete, "_hoodie_commit_seqno", "_hoodie_record_key");
+ assertEquals(hasBuiltInDelete, readerContext.getNeedsBootstrapMerge());
+ } else if (mergeMode == CUSTOM && isProjectionCompatible) {
+ expectedRequiredSchema = generateProjectionSchema("_hoodie_commit_seqno", "_hoodie_record_key", "begin_lat", "begin_lon", "timestamp");
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ } else {
+ expectedRequiredSchema = dataSchema;
+ assertTrue(readerContext.getNeedsBootstrapMerge());
+ }
+ if (supportsParquetRowIndex && mergeUseRecordPosition) {
+ expectedRequiredSchema = addPositionalMergeCol(expectedRequiredSchema);
+ }
+ assertEquals(expectedRequiredSchema, schemaHandler.getRequiredSchema());
+ bootstrapFields = schemaHandler.getBootstrapRequiredFields();
+ expectedBootstrapFields = FileGroupReaderSchemaHandler.getDataAndMetaCols(expectedRequiredSchema);
+ if (supportsParquetRowIndex) {
+ if (!expectedBootstrapFields.getRight().isEmpty()) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ expectedBootstrapFields.getRight().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ } else if (mergeUseRecordPosition) {
+ expectedBootstrapFields.getLeft().add(ParquetRowIndexBasedSchemaHandler.getPositionalMergeField());
+ }
+ }
+ assertEquals(expectedBootstrapFields.getLeft(), bootstrapFields.getLeft());
+ assertEquals(expectedBootstrapFields.getRight(), bootstrapFields.getRight());
+ }
+
+ private static void setupMORTable(RecordMergeMode mergeMode, boolean hasPrecombine, HoodieTableConfig hoodieTableConfig) {
+ when(hoodieTableConfig.populateMetaFields()).thenReturn(Boolean.TRUE);
+ when(hoodieTableConfig.getRecordMergeMode()).thenReturn(mergeMode);
+ if (hasPrecombine) {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn("timestamp");
+ } else {
+ when(hoodieTableConfig.getPreCombineField()).thenReturn(null);
+ }
+ if (mergeMode == CUSTOM) {
+ when(hoodieTableConfig.getRecordMergeStrategyId()).thenReturn("asdf");
+ // NOTE: in this test custom doesn't have any meta cols because it is more interesting of a test case
+ when(hoodieTableConfig.getTableVersion()).thenReturn(HoodieTableVersion.EIGHT);
+ }
+ }
+
+ private static HoodieRecordMerger mockRecordMerger(boolean isProjectionCompatible, String[] mandatoryFields) throws IOException {
+ HoodieRecordMerger merger = mock(HoodieRecordMerger.class);
+ when(merger.isProjectionCompatible()).thenReturn(isProjectionCompatible);
+ when(merger.merge(any(), any(), any(), any(), any())).thenReturn(null);
+ when(merger.getMandatoryFieldsForMerging(any(), any(), any())).thenReturn(mandatoryFields);
+ return merger;
+ }
+
+ static HoodieReaderContext<String> createReaderContext(HoodieTableConfig hoodieTableConfig, boolean supportsParquetRowIndex, boolean hasLogFiles,
+ boolean hasBootstrapBaseFile, boolean mergeUseRecordPosition, HoodieRecordMerger merger) {
+ HoodieReaderContext<String> readerContext = new MockReaderContext(hoodieTableConfig, supportsParquetRowIndex);
+ readerContext.setHasLogFiles(hasLogFiles);
+ readerContext.setHasBootstrapBaseFile(hasBootstrapBaseFile);
+ readerContext.setShouldMergeUseRecordPosition(mergeUseRecordPosition);
+ readerContext.setRecordMerger(Option.ofNullable(merger));
+ return readerContext;
+ }
+
+ static FileGroupReaderSchemaHandler createSchemaHandler(HoodieReaderContext<String> readerContext, Schema dataSchema,
+ Schema requestedSchema, HoodieTableConfig hoodieTableConfig,
+ boolean supportsParquetRowIndex) {
+ return supportsParquetRowIndex
+ ? new ParquetRowIndexBasedSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties())
+ : new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema,
+ Option.empty(), hoodieTableConfig, new TypedProperties());
+ }
+
+ static Schema generateProjectionSchema(String... fields) {
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, Arrays.asList(fields));
+ }
+
+ private static Schema generateProjectionSchema(boolean hasBuiltInDelete, String... fields) {
+ List<String> fieldList = Arrays.asList(fields);
+ if (hasBuiltInDelete) {
+ fieldList = new ArrayList<>(fieldList);
+ fieldList.add("_hoodie_is_deleted");
+ }
+ return HoodieAvroUtils.generateProjectionSchema(DATA_SCHEMA, fieldList);
+ }
+
+ Schema.Field getField(String fieldName) {
+ return DATA_SCHEMA.getField(fieldName);
+ }
+
+ static class MockReaderContext extends HoodieReaderContext<String> {
Review Comment:
Is it possible to swap this for an actual mock? if not, I typically call a class like this "Stubbed" instead of a mock to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3031268164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582)
* 4dd4ea2a3b85d60ff48850130a9336b9df5d244c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3031424485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582)
* 4dd4ea2a3b85d60ff48850130a9336b9df5d244c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3021736712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 9036274b26750e16760b098f6894a42d2ef5f449 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3021667098
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 646f07497bc45fb179cf3c2a7b979f54c13c4239 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532)
* 9036274b26750e16760b098f6894a42d2ef5f449 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3024005624
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 757813fe29f7c10735bfb07d6e35c2f0a624e792 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550)
* 0e6cd937c9c0775fc29ab1c0ae15a265747891b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2779220604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* 95e3598fd2de8830b945234ee25ba328376068af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618)
* 91f7d3c60651beea6b7a99c82132e4d558c20f05 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2776204675
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565)
* e57d02ea630fe7fcedb6de0c2329b4ce9bed716a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614)
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2027342171
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedSchemaHandler.java:
##########
@@ -50,10 +51,18 @@ public HoodiePositionBasedSchemaHandler(HoodieReaderContext<T> readerContext,
super(readerContext, dataSchema, requestedSchema, internalSchemaOpt, hoodieTableConfig, properties);
}
+ private boolean morMergeNeedsPositionCol() {
Review Comment:
outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2769920576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e98e43dd1d1b5fa02c48f0eb91cda80256d7827 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558)
* f2d81ee354f9135c61600ca39f6c85f39a8a9250 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12340:
URL: https://github.com/apache/hudi/pull/12340#discussion_r2023146729
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -165,7 +165,7 @@ class SparkFileFormatInternalRowReaderContext(parquetFileReader: SparkParquetRea
HoodieAvroUtils.removeFields(skeletonRequiredSchema, rowIndexColumn))
//If we need to do position based merging with log files we will leave the row index column at the end
- val dataProjection = if (getHasLogFiles && getShouldMergeUseRecordPosition) {
+ val dataProjection = if (getShouldMergeUseRecordPosition) {
Review Comment:
Part of this pr is to cleanup/clarify what things mean. shouldMergeUseRecordPosition has been clarified to be specific to mor merging, so log files must be present for this to be true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-2560282636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e98e43dd1d1b5fa02c48f0eb91cda80256d7827 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8581] Test schema handler in fg reader and some refactoring to prevent bugs in the future [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12340:
URL: https://github.com/apache/hudi/pull/12340#issuecomment-3078270738
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1983",
"triggerID" : "c4f7b5f7f2b6485fbac119f644699292a1009ed2",
"triggerType" : "PUSH"
}, {
"hash" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2303",
"triggerID" : "c03776da2feacc0426c98079fc6393dc0a6838ff",
"triggerType" : "PUSH"
}, {
"hash" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2558",
"triggerID" : "8e98e43dd1d1b5fa02c48f0eb91cda80256d7827",
"triggerType" : "PUSH"
}, {
"hash" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4565",
"triggerID" : "f2d81ee354f9135c61600ca39f6c85f39a8a9250",
"triggerType" : "PUSH"
}, {
"hash" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4614",
"triggerID" : "e57d02ea630fe7fcedb6de0c2329b4ce9bed716a",
"triggerType" : "PUSH"
}, {
"hash" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c",
"triggerType" : "PUSH"
}, {
"hash" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27d9ba5f21d1ded86846e9bee603361504ce2322",
"triggerType" : "PUSH"
}, {
"hash" : "95e3598fd2de8830b945234ee25ba328376068af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4618",
"triggerID" : "95e3598fd2de8830b945234ee25ba328376068af",
"triggerType" : "PUSH"
}, {
"hash" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4660",
"triggerID" : "91f7d3c60651beea6b7a99c82132e4d558c20f05",
"triggerType" : "PUSH"
}, {
"hash" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6532",
"triggerID" : "646f07497bc45fb179cf3c2a7b979f54c13c4239",
"triggerType" : "PUSH"
}, {
"hash" : "9036274b26750e16760b098f6894a42d2ef5f449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6535",
"triggerID" : "9036274b26750e16760b098f6894a42d2ef5f449",
"triggerType" : "PUSH"
}, {
"hash" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6550",
"triggerID" : "757813fe29f7c10735bfb07d6e35c2f0a624e792",
"triggerType" : "PUSH"
}, {
"hash" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6564",
"triggerID" : "0e6cd937c9c0775fc29ab1c0ae15a265747891b3",
"triggerType" : "PUSH"
}, {
"hash" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6582",
"triggerID" : "11fe4a6b8e0647a77ebe705644a0d3c12ac0abc6",
"triggerType" : "PUSH"
}, {
"hash" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6604",
"triggerID" : "4dd4ea2a3b85d60ff48850130a9336b9df5d244c",
"triggerType" : "PUSH"
}, {
"hash" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631",
"triggerID" : "e420f4735b6fc921050e4e2805c90b2479e937e5",
"triggerType" : "PUSH"
}, {
"hash" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd2744d807dd422b516144894e6d026c34ddd92",
"triggerType" : "PUSH"
}, {
"hash" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0306dba3da80fa63984356185c93d6e6ba9e9b54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f58f8a5bf9f57c11255bc956f0f9cc8746ba4d1c UNKNOWN
* 27d9ba5f21d1ded86846e9bee603361504ce2322 UNKNOWN
* e420f4735b6fc921050e4e2805c90b2479e937e5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6631)
* bbd2744d807dd422b516144894e6d026c34ddd92 UNKNOWN
* 0306dba3da80fa63984356185c93d6e6ba9e9b54 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org