You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/06/09 12:46:39 UTC
[PR] [DRAFT] Standardize compaction and log compaction to use FileGroupReader [hudi]
the-other-tim-brown opened a new pull request, #13411:
URL: https://github.com/apache/hudi/pull/13411
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2978698569
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efce133ff21bb9116d1ad0330e404f150d0f7f41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224)
* 456214637aca7d278a8536c381a5328d07c7f7cc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2140312339
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
@cshuo the meta fields are not populated with values but the output record's schema will have the meta fields.
@danny0405 this is currently [required](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L372) for compaction but does not make a lot of sense to me so I am going to see if I can get rid of this requirement in the new FileGroupReader based flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139103821
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
Not sure why we need this special handling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139103821
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
Not sure why we need this special handling? Please get rid of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2143736659
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
I dug into this a bit, the meta-fields are always present in the writer schema when writing to the table so the outputs of the FGReader for the sake of compaction are always expected to have the meta-fields. In order to avoid unnecessary extra conversion, I am updating the code to directly create the generic record so the merge output is in the expected schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2144120313
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/SortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * An extension of {@link KeyBasedFileGroupRecordBuffer} that sorts the log records based on the record key when joining with the base file records.
+ * This assumes that the base file records are already sorted by the record key.
+ */
+public class SortedKeyBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
+
+ // when sorting is enabled, this will hold the base file record if it was not used in the previous iteration
+ private Option<T> queuedBaseFileRecord = Option.empty();
+ private Queue<String> logRecordKeysSorted = null;
+
+ public SortedKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ HoodieReadStats readStats,
+ Option<String> orderingFieldName,
+ boolean emitDelete) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, props, readStats, orderingFieldName, emitDelete);
+ }
+
+ @Override
+ protected void initializeLogRecordIterator() {
+ logRecordIterator = records.values().stream().sorted(Comparator.comparing(BufferedRecord::getRecordKey)).iterator();
Review Comment:
we can move line 95 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139104149
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java:
##########
@@ -226,7 +226,7 @@ private Object getFieldValueFromIndexedRecord(
IndexedRecord record,
Schema recordSchema,
String fieldName) {
- Schema currentSchema = recordSchema;
+ Schema currentSchema = record.getSchema();
Review Comment:
the `recordShema` is never used now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139799719
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
The rewrite has performance loss also, let's try to avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980781584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bc18e5eaa267a051950bb50653f53e7d0e7d6b0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276)
* 1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279)
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980770170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bc18e5eaa267a051950bb50653f53e7d0e7d6b0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276)
* 1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981620037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2157859992
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -139,23 +137,65 @@ private void init(CompactionOperation operation, String partitionPath, Option<Ho
// Create the writer for writing the new version file
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
- config, writeSchemaWithMetaFields, taskContextSupplier, getRecordType());
+ config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit " + instantTime, io);
+ LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
- protected abstract HoodieRecord.HoodieRecordType getRecordType();
-
/**
* Reads the file slice of a compaction operation using a file group reader,
- * by getting an iterator of the records; then writes the records to a new base file
- * using parquet writer.
+ * by getting an iterator of the records; then writes the records to a new base file.
*/
- protected abstract void write();
+ public void write() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = TypedProperties.copy(config.getProps());
Review Comment:
I see it here in `HoodieFileGroupReader` constructor:
```java
this.orderingFieldName = recordMergeMode == RecordMergeMode.COMMIT_TIME_ORDERING
? Option.empty()
: Option.ofNullable(ConfigUtils.getOrderingField(props))
.or(() -> {
String preCombineField = hoodieTableMetaClient.getTableConfig().getPreCombineField();
if (StringUtils.isNullOrEmpty(preCombineField)) {
return Option.empty();
}
return Option.of(preCombineField);
});
```
So it only affects the old payload and also the merger based merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2145336230
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java:
##########
@@ -205,6 +210,15 @@ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O
return data.collectAsList().stream().parallel().reduce(zeroValue, seqOp::apply, combOp::apply);
}
+ @Override
+ public ReaderContextFactory<?> getReaderContextFactory(HoodieTableMetaClient metaClient) {
+ // metadata table and HFile reads are only supported by the AvroReaderContext.
+ if (metaClient.isMetadataTable() || HFILE.getFileExtension().equals(metaClient.getTableConfig().getBaseFileFormat().getFileExtension())) {
Review Comment:
This only came up in a test, I am not sure if this something we support in practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2146020655
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
if we can assume that the only possibilities for the schema for the special case of metadata payload are the `HoodieMetadataRecord`'s standard schema or that schema with meta-fields appended at the beginning, a simpler check can just be the number of fields being equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151229813
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = new TypedProperties();
+ hoodieTable.getMetaClient().getTableConfig().getProps().forEach(props::putIfAbsent);
Review Comment:
not a blocker.
if we see this pattern repeating in many places where we wanted to construct a new TypedProperties from HoodieWriteconfig and tableConfig, we can make a common utility method.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
I feel we should not leak the payload into to this class.
as of now, we only use `BaseAvroPayload` and no other payloads in here.
I checked the impl of
```
public boolean isDeleted() {
return isDeletedRecord;
}
```
in HoodieMetadataPayload. and wondering why can't we just impl the same method.
```
public boolean isDeleted(Schema schema, Properties props) {
return isDeletedRecord;
}
```
and it should work right?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -173,6 +216,7 @@ public List<WriteStatus> close() {
writeStatus.getStat().setTotalLogBlocks(readStats.getTotalLogBlocks());
writeStatus.getStat().setTotalCorruptLogBlock(readStats.getTotalCorruptLogBlock());
writeStatus.getStat().setTotalRollbackBlocks(readStats.getTotalRollbackBlocks());
+ writeStatus.getStat().setTotalLogSizeCompacted(operation.getMetrics().get(CompactionStrategy.TOTAL_LOG_FILE_SIZE).longValue());
Review Comment:
where are we setting the totalUpsertTime w/n RuntimeStats?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -139,23 +137,68 @@ private void init(CompactionOperation operation, String partitionPath, Option<Ho
// Create the writer for writing the new version file
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
- config, writeSchemaWithMetaFields, taskContextSupplier, getRecordType());
+ config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit " + instantTime, io);
+ LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
- protected abstract HoodieRecord.HoodieRecordType getRecordType();
-
/**
* Reads the file slice of a compaction operation using a file group reader,
* by getting an iterator of the records; then writes the records to a new base file
* using parquet writer.
*/
- protected abstract void write();
+ public void write() {
Review Comment:
minor.
lets fix the java docs for this method. it talks about parquet writer.
but in case of metadata table, its not parquet.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -20,81 +20,80 @@
package org.apache.hudi.io;
import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.engine.TaskContextSupplier;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.CompactionOperation;
import org.apache.hudi.common.model.FileSlice;
import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieLogFile;
import org.apache.hudi.common.model.HoodiePartitionMetadata;
import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.model.HoodieWriteStat;
import org.apache.hudi.common.table.read.HoodieFileGroupReader;
import org.apache.hudi.common.table.read.HoodieReadStats;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
import org.apache.hudi.io.storage.HoodieFileWriterFactory;
-import org.apache.hudi.keygen.BaseKeyGenerator;
import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.table.HoodieTable;
import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import javax.annotation.concurrent.NotThreadSafe;
+
import java.io.IOException;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
-import java.util.stream.Collectors;
+import java.util.Map;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+import static org.apache.hudi.common.model.HoodieFileFormat.HFILE;
/**
- * A base merge handle implementation based on the {@link HoodieFileGroupReader}.
+ * A merge handle implementation based on the {@link HoodieFileGroupReader}.
* <p>
* This merge handle is used for compaction, which passes a file slice from the
* compaction operation of a single file group to a file group reader, get an iterator of
* the records, and writes the records to a new base file.
*/
-public abstract class BaseFileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
- private static final Logger LOG = LoggerFactory.getLogger(BaseFileGroupReaderBasedMergeHandle.class);
-
- protected HoodieReaderContext<T> readerContext;
- protected FileSlice fileSlice;
- protected HoodieReadStats readStats;
-
- public BaseFileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
- CompactionOperation operation, TaskContextSupplier taskContextSupplier,
- Option<BaseKeyGenerator> keyGeneratorOpt,
- HoodieReaderContext<T> readerContext) {
+@NotThreadSafe
+public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
+ private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
+
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private final String maxInstantTime;
+ private HoodieReadStats readStats;
+ private final HoodieRecord.HoodieRecordType recordType;
+
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext, String maxInstantTime,
+ HoodieRecord.HoodieRecordType enginRecordType) {
super(config, instantTime, operation.getPartitionPath(), operation.getFileId(), hoodieTable, taskContextSupplier);
+ this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- Option<HoodieBaseFile> baseFileOpt =
- operation.getBaseFile(config.getBasePath(), operation.getPartitionPath());
- List<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(p ->
- new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), p)))
- .collect(Collectors.toList());
- this.fileSlice = new FileSlice(
- operation.getFileGroupId(),
- operation.getBaseInstantTime(),
- baseFileOpt.isPresent() ? baseFileOpt.get() : null,
- logFiles);
+ this.fileSlice = fileSlice;
+ this.operation = operation;
this.preserveMetadata = true;
- init(operation, this.partitionPath, baseFileOpt);
- validateAndSetAndKeyGenProps(keyGeneratorOpt, config.populateMetaFields());
- }
-
- private void validateAndSetAndKeyGenProps(Option<BaseKeyGenerator> keyGeneratorOpt, boolean populateMetaFields) {
- ValidationUtils.checkArgument(populateMetaFields == !keyGeneratorOpt.isPresent());
- this.keyGeneratorOpt = keyGeneratorOpt;
+ // If the table is a metadata table or the base file is an HFile, we use AVRO record type, otherwise we use the engine record type.
+ this.recordType = hoodieTable.isMetadataTable() || HFILE.getFileExtension().equals(hoodieTable.getBaseFileExtension()) ? HoodieRecord.HoodieRecordType.AVRO : enginRecordType;
Review Comment:
instead of fixing this here, shouldn't we fix
HoodieCompactor.getEngineRecordType() to return the right value.
we might have to rename the method though if we fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981986456
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d56bfb165003beb622e39d63a0970a43479beb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2153243816
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
sg then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981715560
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286)
* 64d56bfb165003beb622e39d63a0970a43479beb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2960289134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086)
* 6a86f9effc3fa30f4006da2e44a1da6e4f68a222 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139106082
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
I guess this is because the payload merged has schema discrepencies. cc @cshuo for confirming this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139106993
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FlinkReaderContextFactory.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+
+import java.util.Collections;
+import java.util.function.Supplier;
+
+public class FlinkReaderContextFactory implements ReaderContextFactory<RowData> {
Review Comment:
Add some doc for it, it should only be used in table services: `compaction`, `clustering`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139125531
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
+ }
}
@Override
public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
- return getInsertValue(schema, new Properties());
+ return getInsertValue(schema, null);
Review Comment:
I can use that but it is actually a mutable, shared instance. That means if any user modifies the properties, they will get leaked to other sites. The code needs to be fixed to make that instance immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2960596322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a86f9effc3fa30f4006da2e44a1da6e4f68a222 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2971499278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178)
* 59444a040c40c4e03a5af12049a5000c049773f0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975123415
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f0ae0a1fc83618d08e9278394253f18af58c176 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222)
* efce133ff21bb9116d1ad0330e404f150d0f7f41 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151261114
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -173,6 +216,7 @@ public List<WriteStatus> close() {
writeStatus.getStat().setTotalLogBlocks(readStats.getTotalLogBlocks());
writeStatus.getStat().setTotalCorruptLogBlock(readStats.getTotalCorruptLogBlock());
writeStatus.getStat().setTotalRollbackBlocks(readStats.getTotalRollbackBlocks());
+ writeStatus.getStat().setTotalLogSizeCompacted(operation.getMetrics().get(CompactionStrategy.TOTAL_LOG_FILE_SIZE).longValue());
Review Comment:
It currently happens in the `close` method of `HoodieMergeHandle` which is called above with `super.close()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2152308434
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +380,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HOODIE_METADATA_SCHEMA == schema) {
+ // If the schema is same or none is provided, we can return the record directly
+ HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
+ columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
+ return Option.of(record);
+ } else {
+ // Otherwise, the assumption is that the schema required contains the metadata fields so we construct a new GenericRecord with these fields
+ GenericData.Record record = new GenericData.Record(schema);
+ int offset = schema.getField("key").pos();
+ record.put(offset, key);
+ record.put(offset + 1, type);
+ if (filesystemMetadata != null) {
Review Comment:
Right now we're doing some optimization by assuming the schema is always cached so we can use `==`, if we can be sure all paths for metadata table are tested then we can hunt down any places skipping the cache. Otherwise, we don't know if the schema is not the same instance because it is coming from a different code path or because it has the metadata fields added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2982461745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
}, {
"hash" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d56bfb165003beb622e39d63a0970a43479beb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290)
* 0a4449c6b7ef27225850795aee64cd1d4e6fbf52 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980776294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bc18e5eaa267a051950bb50653f53e7d0e7d6b0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276)
* 1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13411:
URL: https://github.com/apache/hudi/pull/13411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2143615810
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
The optimization happens when checking if two schemas are equal to each other. Caching the generated version means the equality check can pass on checking if it is the same instance instead of inspecting all the fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975041989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59444a040c40c4e03a5af12049a5000c049773f0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199)
* 9f0ae0a1fc83618d08e9278394253f18af58c176 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2978616352
+1 for the nice work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2147933825
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
Yes for compaction this will work but it does not work in any of the cases where the meta fields are not part of the input schema. This will happen when first writing to the metadata table if you are looking for such a case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2148900762
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
Review Comment:
Seems `PositionBasedFileGroupRecordBuffer` has a fallback checking if there is no record positions in data block, so checking for engine type is not needed actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981710878
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286)
* 64d56bfb165003beb622e39d63a0970a43479beb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2153249062
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -20,81 +20,80 @@
package org.apache.hudi.io;
import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.engine.TaskContextSupplier;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.CompactionOperation;
import org.apache.hudi.common.model.FileSlice;
import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieLogFile;
import org.apache.hudi.common.model.HoodiePartitionMetadata;
import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.model.HoodieWriteStat;
import org.apache.hudi.common.table.read.HoodieFileGroupReader;
import org.apache.hudi.common.table.read.HoodieReadStats;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
import org.apache.hudi.io.storage.HoodieFileWriterFactory;
-import org.apache.hudi.keygen.BaseKeyGenerator;
import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.table.HoodieTable;
import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import javax.annotation.concurrent.NotThreadSafe;
+
import java.io.IOException;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
-import java.util.stream.Collectors;
+import java.util.Map;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+import static org.apache.hudi.common.model.HoodieFileFormat.HFILE;
/**
- * A base merge handle implementation based on the {@link HoodieFileGroupReader}.
+ * A merge handle implementation based on the {@link HoodieFileGroupReader}.
* <p>
* This merge handle is used for compaction, which passes a file slice from the
* compaction operation of a single file group to a file group reader, get an iterator of
* the records, and writes the records to a new base file.
*/
-public abstract class BaseFileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
- private static final Logger LOG = LoggerFactory.getLogger(BaseFileGroupReaderBasedMergeHandle.class);
-
- protected HoodieReaderContext<T> readerContext;
- protected FileSlice fileSlice;
- protected HoodieReadStats readStats;
-
- public BaseFileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
- CompactionOperation operation, TaskContextSupplier taskContextSupplier,
- Option<BaseKeyGenerator> keyGeneratorOpt,
- HoodieReaderContext<T> readerContext) {
+@NotThreadSafe
+public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
+ private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
+
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private final String maxInstantTime;
+ private HoodieReadStats readStats;
+ private final HoodieRecord.HoodieRecordType recordType;
+
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext, String maxInstantTime,
+ HoodieRecord.HoodieRecordType enginRecordType) {
super(config, instantTime, operation.getPartitionPath(), operation.getFileId(), hoodieTable, taskContextSupplier);
+ this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- Option<HoodieBaseFile> baseFileOpt =
- operation.getBaseFile(config.getBasePath(), operation.getPartitionPath());
- List<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(p ->
- new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), p)))
- .collect(Collectors.toList());
- this.fileSlice = new FileSlice(
- operation.getFileGroupId(),
- operation.getBaseInstantTime(),
- baseFileOpt.isPresent() ? baseFileOpt.get() : null,
- logFiles);
+ this.fileSlice = fileSlice;
+ this.operation = operation;
this.preserveMetadata = true;
- init(operation, this.partitionPath, baseFileOpt);
- validateAndSetAndKeyGenProps(keyGeneratorOpt, config.populateMetaFields());
- }
-
- private void validateAndSetAndKeyGenProps(Option<BaseKeyGenerator> keyGeneratorOpt, boolean populateMetaFields) {
- ValidationUtils.checkArgument(populateMetaFields == !keyGeneratorOpt.isPresent());
- this.keyGeneratorOpt = keyGeneratorOpt;
+ // If the table is a metadata table or the base file is an HFile, we use AVRO record type, otherwise we use the engine record type.
+ this.recordType = hoodieTable.isMetadataTable() || HFILE.getFileExtension().equals(hoodieTable.getBaseFileExtension()) ? HoodieRecord.HoodieRecordType.AVRO : enginRecordType;
Review Comment:
sg. if this is the only use of deducing record type, its fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2961158129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a86f9effc3fa30f4006da2e44a1da6e4f68a222 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108)
* de94df1ece1dab7a903098a56a44eb305f24b801 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139104149
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java:
##########
@@ -226,7 +226,7 @@ private Object getFieldValueFromIndexedRecord(
IndexedRecord record,
Schema recordSchema,
String fieldName) {
- Schema currentSchema = recordSchema;
+ Schema currentSchema = record.getSchema();
Review Comment:
the record shema is never used now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2961114643
cc @cshuo for reviewing Flink related changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139121781
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -121,6 +129,18 @@ public boolean containsLogRecord(String recordKey) {
protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
String recordKey = readerContext.getRecordKey(baseRecord, readerSchema);
+ int comparison;
+ while (sortOutput && !getLogRecordKeysSorted().isEmpty() && (comparison = getLogRecordKeysSorted().peek().compareTo(recordKey)) <= 0) {
Review Comment:
The sorting is also required in the merge handle so it cannot go into the `FileGroupReaderBasedAppendHandle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2971727599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59444a040c40c4e03a5af12049a5000c049773f0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968106503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de94df1ece1dab7a903098a56a44eb305f24b801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120)
* 02959b1adf82b4b8660965ac9b11101871ca8583 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975260239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efce133ff21bb9116d1ad0330e404f150d0f7f41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2971502222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178)
* 59444a040c40c4e03a5af12049a5000c049773f0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2146281427
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java:
##########
@@ -205,6 +210,15 @@ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O
return data.collectAsList().stream().parallel().reduce(zeroValue, seqOp::apply, combOp::apply);
}
+ @Override
+ public ReaderContextFactory<?> getReaderContextFactory(HoodieTableMetaClient metaClient) {
+ // metadata table and HFile reads are only supported by the AvroReaderContext.
+ if (metaClient.isMetadataTable() || HFILE.getFileExtension().equals(metaClient.getTableConfig().getBaseFileFormat().getFileExtension())) {
Review Comment:
I don't think it is supported, the test should be aming to test the MDT instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2146280368
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
I don't think we need that, just intern the schema on the fly where necessary because it's low frequency invocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139122756
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
Yes, the merging of two log file records does not inspect the output schema in the FileGroupRecordBuffer so there is a possibility that the output schema is incorrect without this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2961159794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a86f9effc3fa30f4006da2e44a1da6e4f68a222 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108)
* de94df1ece1dab7a903098a56a44eb305f24b801 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968301531
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02959b1adf82b4b8660965ac9b11101871ca8583 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174)
* 4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139105703
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java:
##########
@@ -121,6 +129,18 @@ public boolean containsLogRecord(String recordKey) {
protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
String recordKey = readerContext.getRecordKey(baseRecord, readerSchema);
+ int comparison;
+ while (sortOutput && !getLogRecordKeysSorted().isEmpty() && (comparison = getLogRecordKeysSorted().peek().compareTo(recordKey)) <= 0) {
Review Comment:
either we have a separate record buffer impl (by sub-class `KeyBasedFileGroupRecordBuffer`) or move the sorting into `FileGroupReaderBasedAppendHandle` itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151268300
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = new TypedProperties();
+ hoodieTable.getMetaClient().getTableConfig().getProps().forEach(props::putIfAbsent);
Review Comment:
This code can be simplified, we don't need the table config in the properties object for the FileGroupReader. It is used to get the configs for the spillable map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980503032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 456214637aca7d278a8536c381a5328d07c7f7cc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253)
* 2bc18e5eaa267a051950bb50653f53e7d0e7d6b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981196260
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2982634691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
}, {
"hash" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6306",
"triggerID" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0a4449c6b7ef27225850795aee64cd1d4e6fbf52 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6306)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980990119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279)
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139106495
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
+ }
}
@Override
public Option<IndexedRecord> getInsertValue(Schema schema) throws IOException {
- return getInsertValue(schema, new Properties());
+ return getInsertValue(schema, null);
Review Comment:
`CollectionUtils.EMPTY_PROPERTIES` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975043875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59444a040c40c4e03a5af12049a5000c049773f0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199)
* 9f0ae0a1fc83618d08e9278394253f18af58c176 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980508833
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 456214637aca7d278a8536c381a5328d07c7f7cc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253)
* 2bc18e5eaa267a051950bb50653f53e7d0e7d6b0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2155746951
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = TypedProperties.copy(config.getProps());
+ long maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(taskContextSupplier, config);
+ props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
Review Comment:
Is the orderingValue set up correctly for merger based merging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2152313301
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/SortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * An extension of {@link KeyBasedFileGroupRecordBuffer} that sorts the log records based on the record key when joining with the base file records.
+ * This assumes that the base file records are already sorted by the record key.
+ */
+public class SortedKeyBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
+
+ // when sorting is enabled, this will hold the base file record if it was not used in the previous iteration
+ private Option<T> queuedBaseFileRecord = Option.empty();
+ private Queue<String> logRecordKeysSorted = null;
+
+ public SortedKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ HoodieReadStats readStats,
+ Option<String> orderingFieldName,
+ boolean emitDelete) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, props, readStats, orderingFieldName, emitDelete);
+ }
+
+ @Override
+ protected void initializeLogRecordIterator() {
+ logRecordIterator = records.values().stream().sorted(Comparator.comparing(BufferedRecord::getRecordKey)).iterator();
+ }
+
+ @Override
+ protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
+ String recordKey = readerContext.getRecordKey(baseRecord, readerSchema);
+ int comparison;
+ while (!getLogRecordKeysSorted().isEmpty() && (comparison = getLogRecordKeysSorted().peek().compareTo(recordKey)) <= 0) {
+ String nextLogRecordKey = getLogRecordKeysSorted().poll();
+ if (comparison < 0) {
+ BufferedRecord<T> nextLogRecord = records.remove(nextLogRecordKey);
+ if (!nextLogRecord.isDelete() || emitDelete) {
+ nextRecord = nextLogRecord.getRecord();
Review Comment:
Yes, will add some inline logs here to cover the cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2981617895
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968244509
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02959b1adf82b4b8660965ac9b11101871ca8583 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2147396418
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
If we are sure the MDT payload reading from log file always have metadata fields, using `HoodieMetadataRecord.getClassSchema() == schema` should be working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2145365865
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/SortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * An extension of {@link KeyBasedFileGroupRecordBuffer} that sorts the log records based on the record key when joining with the base file records.
+ * This assumes that the base file records are already sorted by the record key.
+ */
+public class SortedKeyBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
+
+ // when sorting is enabled, this will hold the base file record if it was not used in the previous iteration
+ private Option<T> queuedBaseFileRecord = Option.empty();
+ private Queue<String> logRecordKeysSorted = null;
+
+ public SortedKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ HoodieReadStats readStats,
+ Option<String> orderingFieldName,
+ boolean emitDelete) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, props, readStats, orderingFieldName, emitDelete);
+ }
+
+ @Override
+ protected void initializeLogRecordIterator() {
+ logRecordIterator = records.values().stream().sorted(Comparator.comparing(BufferedRecord::getRecordKey)).iterator();
Review Comment:
The `logRecordIterator` is only initialized once we've iterated through all the base file records so it cannot be initialized with the queue of log record keys. One option is to create a queue with the records but that will force us to pull all log records into memory up front instead of relying on spilling to disk in the case of many log records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975056707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f0ae0a1fc83618d08e9278394253f18af58c176 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222)
* efce133ff21bb9116d1ad0330e404f150d0f7f41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2143252175
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
Review Comment:
Is this just a note or do we need to add a check for engine type here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2144140166
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
if we are confident the schema has been interned, use `==` would be more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2152005788
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +380,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HOODIE_METADATA_SCHEMA == schema) {
+ // If the schema is same or none is provided, we can return the record directly
+ HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
+ columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
+ return Option.of(record);
+ } else {
+ // Otherwise, the assumption is that the schema required contains the metadata fields so we construct a new GenericRecord with these fields
+ GenericData.Record record = new GenericData.Record(schema);
+ int offset = schema.getField("key").pos();
+ record.put(offset, key);
+ record.put(offset + 1, type);
+ if (filesystemMetadata != null) {
Review Comment:
if `offset` is a constant itself, we can use all the constants directly for the fields here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139121122
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
This is an optimization so the interned schema is always the same as the record schema provided by the generated java classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151257632
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -20,81 +20,80 @@
package org.apache.hudi.io;
import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.engine.TaskContextSupplier;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.CompactionOperation;
import org.apache.hudi.common.model.FileSlice;
import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieLogFile;
import org.apache.hudi.common.model.HoodiePartitionMetadata;
import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.model.HoodieWriteStat;
import org.apache.hudi.common.table.read.HoodieFileGroupReader;
import org.apache.hudi.common.table.read.HoodieReadStats;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
import org.apache.hudi.io.storage.HoodieFileWriterFactory;
-import org.apache.hudi.keygen.BaseKeyGenerator;
import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.table.HoodieTable;
import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import javax.annotation.concurrent.NotThreadSafe;
+
import java.io.IOException;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
-import java.util.stream.Collectors;
+import java.util.Map;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+import static org.apache.hudi.common.model.HoodieFileFormat.HFILE;
/**
- * A base merge handle implementation based on the {@link HoodieFileGroupReader}.
+ * A merge handle implementation based on the {@link HoodieFileGroupReader}.
* <p>
* This merge handle is used for compaction, which passes a file slice from the
* compaction operation of a single file group to a file group reader, get an iterator of
* the records, and writes the records to a new base file.
*/
-public abstract class BaseFileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
- private static final Logger LOG = LoggerFactory.getLogger(BaseFileGroupReaderBasedMergeHandle.class);
-
- protected HoodieReaderContext<T> readerContext;
- protected FileSlice fileSlice;
- protected HoodieReadStats readStats;
-
- public BaseFileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
- CompactionOperation operation, TaskContextSupplier taskContextSupplier,
- Option<BaseKeyGenerator> keyGeneratorOpt,
- HoodieReaderContext<T> readerContext) {
+@NotThreadSafe
+public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieMergeHandle<T, I, K, O> {
+ private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
+
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private final String maxInstantTime;
+ private HoodieReadStats readStats;
+ private final HoodieRecord.HoodieRecordType recordType;
+
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext, String maxInstantTime,
+ HoodieRecord.HoodieRecordType enginRecordType) {
super(config, instantTime, operation.getPartitionPath(), operation.getFileId(), hoodieTable, taskContextSupplier);
+ this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- Option<HoodieBaseFile> baseFileOpt =
- operation.getBaseFile(config.getBasePath(), operation.getPartitionPath());
- List<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(p ->
- new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), p)))
- .collect(Collectors.toList());
- this.fileSlice = new FileSlice(
- operation.getFileGroupId(),
- operation.getBaseInstantTime(),
- baseFileOpt.isPresent() ? baseFileOpt.get() : null,
- logFiles);
+ this.fileSlice = fileSlice;
+ this.operation = operation;
this.preserveMetadata = true;
- init(operation, this.partitionPath, baseFileOpt);
- validateAndSetAndKeyGenProps(keyGeneratorOpt, config.populateMetaFields());
- }
-
- private void validateAndSetAndKeyGenProps(Option<BaseKeyGenerator> keyGeneratorOpt, boolean populateMetaFields) {
- ValidationUtils.checkArgument(populateMetaFields == !keyGeneratorOpt.isPresent());
- this.keyGeneratorOpt = keyGeneratorOpt;
+ // If the table is a metadata table or the base file is an HFile, we use AVRO record type, otherwise we use the engine record type.
+ this.recordType = hoodieTable.isMetadataTable() || HFILE.getFileExtension().equals(hoodieTable.getBaseFileExtension()) ? HoodieRecord.HoodieRecordType.AVRO : enginRecordType;
Review Comment:
We may be able to but it will just be moving this logic into that method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151281331
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/SortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.RecordMergeMode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * An extension of {@link KeyBasedFileGroupRecordBuffer} that sorts the log records based on the record key when joining with the base file records.
+ * This assumes that the base file records are already sorted by the record key.
+ */
+public class SortedKeyBasedFileGroupRecordBuffer<T> extends KeyBasedFileGroupRecordBuffer<T> {
+
+ // when sorting is enabled, this will hold the base file record if it was not used in the previous iteration
+ private Option<T> queuedBaseFileRecord = Option.empty();
+ private Queue<String> logRecordKeysSorted = null;
+
+ public SortedKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ RecordMergeMode recordMergeMode,
+ TypedProperties props,
+ HoodieReadStats readStats,
+ Option<String> orderingFieldName,
+ boolean emitDelete) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, props, readStats, orderingFieldName, emitDelete);
+ }
+
+ @Override
+ protected void initializeLogRecordIterator() {
+ logRecordIterator = records.values().stream().sorted(Comparator.comparing(BufferedRecord::getRecordKey)).iterator();
+ }
+
+ @Override
+ protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
+ String recordKey = readerContext.getRecordKey(baseRecord, readerSchema);
+ int comparison;
+ while (!getLogRecordKeysSorted().isEmpty() && (comparison = getLogRecordKeysSorted().peek().compareTo(recordKey)) <= 0) {
+ String nextLogRecordKey = getLogRecordKeysSorted().poll();
+ if (comparison < 0) {
+ BufferedRecord<T> nextLogRecord = records.remove(nextLogRecordKey);
+ if (!nextLogRecord.isDelete() || emitDelete) {
+ nextRecord = nextLogRecord.getRecord();
Review Comment:
can we add java docs here.
mainly to callout, here we are handling insert records in log files which are not present in base files.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +380,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HOODIE_METADATA_SCHEMA == schema) {
+ // If the schema is same or none is provided, we can return the record directly
+ HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
+ columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
+ return Option.of(record);
+ } else {
+ // Otherwise, the assumption is that the schema required contains the metadata fields so we construct a new GenericRecord with these fields
+ GenericData.Record record = new GenericData.Record(schema);
+ int offset = schema.getField("key").pos();
+ record.put(offset, key);
+ record.put(offset + 1, type);
+ if (filesystemMetadata != null) {
Review Comment:
similar to how we have HoodieRecord.HOODIE_META_COLUMNS_NAME_TO_POS
can we add similar static mapping for metadata table field positions. and use it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2149972782
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
I've updated the code in this [commit](https://github.com/apache/hudi/pull/13411/commits/9f0ae0a1fc83618d08e9278394253f18af58c176) to use the cache in a couple more places so this check can be used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2152005788
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +380,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HOODIE_METADATA_SCHEMA == schema) {
+ // If the schema is same or none is provided, we can return the record directly
+ HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
+ columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
+ return Option.of(record);
+ } else {
+ // Otherwise, the assumption is that the schema required contains the metadata fields so we construct a new GenericRecord with these fields
+ GenericData.Record record = new GenericData.Record(schema);
+ int offset = schema.getField("key").pos();
+ record.put(offset, key);
+ record.put(offset + 1, type);
+ if (filesystemMetadata != null) {
Review Comment:
if `offset` is a constant itself, we can use all the constants directly for the fields here.(the map lookup is also a cost especially when the map key is a string).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2144111794
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java:
##########
@@ -205,6 +210,15 @@ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O
return data.collectAsList().stream().parallel().reduce(zeroValue, seqOp::apply, combOp::apply);
}
+ @Override
+ public ReaderContextFactory<?> getReaderContextFactory(HoodieTableMetaClient metaClient) {
+ // metadata table and HFile reads are only supported by the AvroReaderContext.
+ if (metaClient.isMetadataTable() || HFILE.getFileExtension().equals(metaClient.getTableConfig().getBaseFileFormat().getFileExtension())) {
Review Comment:
in which case the data table cound have `HFile` as the base file format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139374889
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
Yes, schema of merged result from two MDT log file records is always `HoodieMetadataRecord#SCHEMA$`, there is a potential problem that the records in the final iterator can have different schemas.
IIUC, there is problem using `rewriteRecord(record, schema)`, since `record` here is `HoodieMetadataRecord` which is instance of `SpecificRecordBase`, and rewriting will skip metadata fields in new schema. See details in https://github.com/apache/hudi/blob/5146dc81f493b4daa06ee4beee21a4f045ad5718/hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java#L953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2155748129
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -139,23 +137,65 @@ private void init(CompactionOperation operation, String partitionPath, Option<Ho
// Create the writer for writing the new version file
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
- config, writeSchemaWithMetaFields, taskContextSupplier, getRecordType());
+ config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit " + instantTime, io);
+ LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
- protected abstract HoodieRecord.HoodieRecordType getRecordType();
-
/**
* Reads the file slice of a compaction operation using a file group reader,
- * by getting an iterator of the records; then writes the records to a new base file
- * using parquet writer.
+ * by getting an iterator of the records; then writes the records to a new base file.
*/
- protected abstract void write();
+ public void write() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = TypedProperties.copy(config.getProps());
Review Comment:
Is the orderingValue set up correctly for merger based merging?
Please check it:
```java
public static String getOrderingField(Properties properties) {
String orderField = null;
if (properties.containsKey(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP_KEY)) {
orderField = properties.getProperty(HoodiePayloadProps.PAYLOAD_ORDERING_FIELD_PROP_KEY);
} else if (properties.containsKey("hoodie.datasource.write.precombine.field")) {
orderField = properties.getProperty("hoodie.datasource.write.precombine.field");
} else if (properties.containsKey(HoodieTableConfig.PRECOMBINE_FIELD.key())) {
orderField = properties.getProperty(HoodieTableConfig.PRECOMBINE_FIELD.key());
}
return orderField;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2147396184
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java:
##########
@@ -205,6 +210,15 @@ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O
return data.collectAsList().stream().parallel().reduce(zeroValue, seqOp::apply, combOp::apply);
}
+ @Override
+ public ReaderContextFactory<?> getReaderContextFactory(HoodieTableMetaClient metaClient) {
+ // metadata table and HFile reads are only supported by the AvroReaderContext.
+ if (metaClient.isMetadataTable() || HFILE.getFileExtension().equals(metaClient.getTableConfig().getBaseFileFormat().getFileExtension())) {
Review Comment:
yeah, let's update the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2148881859
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
`HoodieMetadataRecord.getClassSchema()` will return an instance of an object but parsing a string of the schema will not return the same exact instance. That was the reasoning for the special handling in the cache schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968109516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de94df1ece1dab7a903098a56a44eb305f24b801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120)
* 02959b1adf82b4b8660965ac9b11101871ca8583 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2145340847
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
`==` would require the optimization I previously made or something similar. The class schema will not match the interned schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2148871388
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +379,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
Review Comment:
> This will happen when first writing to the metadata table if you are looking for such a case.
then the schema should be just the global static schema of the class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2155748129
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -139,23 +137,65 @@ private void init(CompactionOperation operation, String partitionPath, Option<Ho
// Create the writer for writing the new version file
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
- config, writeSchemaWithMetaFields, taskContextSupplier, getRecordType());
+ config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit " + instantTime, io);
+ LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
- protected abstract HoodieRecord.HoodieRecordType getRecordType();
-
/**
* Reads the file slice of a compaction operation using a file group reader,
- * by getting an iterator of the records; then writes the records to a new base file
- * using parquet writer.
+ * by getting an iterator of the records; then writes the records to a new base file.
*/
- protected abstract void write();
+ public void write() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
+ TypedProperties props = TypedProperties.copy(config.getProps());
Review Comment:
Is the orderingValue set up correctly for merger based merging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2153247124
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +380,44 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
- HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
- columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HOODIE_METADATA_SCHEMA == schema) {
+ // If the schema is same or none is provided, we can return the record directly
+ HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
+ columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
+ return Option.of(record);
+ } else {
+ // Otherwise, the assumption is that the schema required contains the metadata fields so we construct a new GenericRecord with these fields
+ GenericData.Record record = new GenericData.Record(schema);
+ int offset = schema.getField("key").pos();
+ record.put(offset, key);
+ record.put(offset + 1, type);
+ if (filesystemMetadata != null) {
Review Comment:
changes lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2982125512
Still test failure:
```java
TestHiveSyncTool.testSyncWithCommentedSchema
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2982464211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6281",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6286",
"triggerID" : "2981617895",
"triggerType" : "MANUAL"
}, {
"hash" : "64d56bfb165003beb622e39d63a0970a43479beb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290",
"triggerID" : "64d56bfb165003beb622e39d63a0970a43479beb",
"triggerType" : "PUSH"
}, {
"hash" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6306",
"triggerID" : "0a4449c6b7ef27225850795aee64cd1d4e6fbf52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64d56bfb165003beb622e39d63a0970a43479beb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6290)
* 0a4449c6b7ef27225850795aee64cd1d4e6fbf52 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6306)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139138306
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FlinkReaderContextFactory.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+
+import java.util.Collections;
+import java.util.function.Supplier;
+
+public class FlinkReaderContextFactory implements ReaderContextFactory<RowData> {
Review Comment:
Added a short doc, let me know if there is more to add to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139374889
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
Yes, schema of merged result from two MDT log file records is always `HoodieMetadataRecord#SCHEMA$`, there is a potential problem that the records in the final iterator can have different schemas.
IIUC, there is problem using `rewriteRecord(record, schema)`, since `record` here is `HoodieMetadataRecord` which is instance of `SpecificRecordBase`, and rewriting will skip metadata fields in new schema. See details in https://github.com/apache/hudi/blob/5146dc81f493b4daa06ee4beee21a4f045ad5718/hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java#L953 Is the skipping as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2960284764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086)
* 6a86f9effc3fa30f4006da2e44a1da6e4f68a222 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2145334842
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
Yes, it's just an optimization so if they are equivalent they will not need to perform the rest of the equals. If we need it we can do something similar for registering initial versions of our predefined schemas in a separate change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2978696258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efce133ff21bb9116d1ad0330e404f150d0f7f41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224)
* 456214637aca7d278a8536c381a5328d07c7f7cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2978789813
Oops. there are test failures:
```java
Error: Errors:
Error: TestHoodieFileSystemViews.testFileSystemViewConsistency:152->upsertRecords:356 ยป HoodieUpsert
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2978795645
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 456214637aca7d278a8536c381a5328d07c7f7cc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2152349565
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
I've been looking for a good place to put this interface and could not find a good one without too much refactoring. I agree with @danny0405 that we should do this when refactoring when deprecating payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2955698652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2147384415
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java:
##########
@@ -205,6 +210,15 @@ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O
return data.collectAsList().stream().parallel().reduce(zeroValue, seqOp::apply, combOp::apply);
}
+ @Override
+ public ReaderContextFactory<?> getReaderContextFactory(HoodieTableMetaClient metaClient) {
+ // metadata table and HFile reads are only supported by the AvroReaderContext.
+ if (metaClient.isMetadataTable() || HFILE.getFileExtension().equals(metaClient.getTableConfig().getBaseFileFormat().getFileExtension())) {
Review Comment:
The test is `TestHoodieSparkMergeOnReadTableInsertUpdateDelete#testSimpleInsertAndUpdate`, the case for HFile will need to be removed if we remove this check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139106082
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
I guess this is because the payload merged has schema discrepencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139192678
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
The `#intern` is low frequency action, there is no much agains here, let's keep the cache working in high-level sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2961294641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de94df1ece1dab7a903098a56a44eb305f24b801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139336935
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.CompactionOperation;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.table.read.HoodieFileGroupReader;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
+
+/**
+ * A base append handle implementation based on the {@link HoodieFileGroupReader}.
+ * <p>
+ * This append-handle is used for log-compaction, which passes a file slice from the
+ * compaction operation of a single file group to a file group reader, get an iterator of
+ * the records, and writes the records to a new log file.
+ */
+@NotThreadSafe
+public class FileGroupReaderBasedAppendHandle<T, I, K, O> extends HoodieAppendHandle<T, I, K, O> {
+ private final HoodieReaderContext<T> readerContext;
+ private final FileSlice fileSlice;
+ private final CompactionOperation operation;
+ private HoodieReadStats readStats;
+
+ public FileGroupReaderBasedAppendHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ FileSlice fileSlice, CompactionOperation operation, TaskContextSupplier taskContextSupplier,
+ HoodieReaderContext<T> readerContext) {
+ super(config, instantTime, hoodieTable, operation.getPartitionPath(), operation.getFileId(), taskContextSupplier);
+ this.operation = operation;
+ this.readerContext = readerContext;
+ this.fileSlice = fileSlice;
+ }
+
+ @Override
+ public void doAppend() {
+ boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
Review Comment:
For flink compaction, `usePosition` always be false.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -20,36 +20,42 @@
package org.apache.hudi.io;
import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.HoodieMemoryConfig;
+import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.engine.HoodieReaderContext;
import org.apache.hudi.common.engine.TaskContextSupplier;
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.model.CompactionOperation;
import org.apache.hudi.common.model.FileSlice;
import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieLogFile;
import org.apache.hudi.common.model.HoodiePartitionMetadata;
import org.apache.hudi.common.model.HoodieRecord;
import org.apache.hudi.common.model.HoodieWriteStat;
import org.apache.hudi.common.table.read.HoodieFileGroupReader;
import org.apache.hudi.common.table.read.HoodieReadStats;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.exception.HoodieUpsertException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.SerDeHelper;
import org.apache.hudi.io.storage.HoodieFileWriterFactory;
-import org.apache.hudi.keygen.BaseKeyGenerator;
import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.table.HoodieTable;
import org.apache.hudi.table.action.compact.strategy.CompactionStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import javax.annotation.concurrent.NotThreadSafe;
+
import java.io.IOException;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
-import java.util.stream.Collectors;
+import java.util.Map;
+
+import static org.apache.hudi.common.config.HoodieReaderConfig.MERGE_USE_RECORD_POSITIONS;
/**
* A base merge handle implementation based on the {@link HoodieFileGroupReader}.
Review Comment:
'base' can be removed here.
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/compact/HoodieFlinkMergeOnReadTableCompactor.java:
##########
@@ -73,10 +73,11 @@ public List<WriteStatus> compact(HoodieCompactionHandler compactionHandler,
CompactionOperation operation,
String instantTime,
TaskContextSupplier taskContextSupplier,
- Option<HoodieReaderContext<?>> readerContextOpt) throws IOException {
+ Option<HoodieReaderContext<?>> readerContextOpt,
+ HoodieTable table) throws IOException {
Review Comment:
If parameter`table` is necessary, `metaClient` in above can be removed and using `table.getMetaClient`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2139374889
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -376,19 +377,24 @@ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord oldRecord, S
}
@Override
- public Option<IndexedRecord> getInsertValue(Schema schemaIgnored, Properties propertiesIgnored) throws IOException {
+ public Option<IndexedRecord> getInsertValue(Schema schema, Properties propertiesIgnored) throws IOException {
if (key == null || this.isDeletedRecord) {
return Option.empty();
}
HoodieMetadataRecord record = new HoodieMetadataRecord(key, type, filesystemMetadata, bloomFilterMetadata,
columnStatMetadata, recordIndexMetadata, secondaryIndexMetadata);
- return Option.of(record);
+ if (schema == null || HoodieMetadataRecord.getClassSchema().equals(schema)) {
+ // If the schema is same or none is provided, we can return the record directly
+ return Option.of(record);
+ } else {
+ return Option.of(rewriteRecord(record, schema));
Review Comment:
Yes, schema of merged result from two MDT log file records is always `HoodieMetadataRecord#SCHEMA$`, there is a potential problem that the records in the final iterator can have different schemas.
IIUC, there is problem using `rewriteRecord(record, schema)`, since `record` here is `HoodieMetadataRecord` which is instance of `SpecificRecordBase`, and rewriting will skip metadata fields in new schema. See details in https://github.com/apache/hudi/blob/5146dc81f493b4daa06ee4beee21a4f045ad5718/hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java#L953. Is the skipping as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968306005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02959b1adf82b4b8660965ac9b11101871ca8583 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174)
* 4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2968451309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2144108759
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaCache.java:
##########
@@ -33,7 +35,12 @@ public class AvroSchemaCache {
// Ensure that there is only one variable instance of the same schema within an entire JVM lifetime
- private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(k -> k);
+ private static final LoadingCache<Schema, Schema> SCHEMA_CACHE = Caffeine.newBuilder().weakValues().maximumSize(1024).build(inputSchema -> {
+ if (inputSchema.equals(HoodieMetadataRecord.getClassSchema())) {
Review Comment:
Did't get your point, the `Schema#equals` will firstly check the reference equals anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2975054812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59444a040c40c4e03a5af12049a5000c049773f0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199)
* 9f0ae0a1fc83618d08e9278394253f18af58c176 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222)
* efce133ff21bb9116d1ad0330e404f150d0f7f41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151259532
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
Are you suggesting adding a new method to the interface for payloads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151408016
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
If `HoodieMetadataPayload` and `BaseAvroPayload` share the same interface, then we can simplify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13411:
URL: https://github.com/apache/hudi/pull/13411#issuecomment-2980787001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6086",
"triggerID" : "7ac83aab43d92a0a95efe6a9bf5d199c9bc4599c",
"triggerType" : "PUSH"
}, {
"hash" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6108",
"triggerID" : "6a86f9effc3fa30f4006da2e44a1da6e4f68a222",
"triggerType" : "PUSH"
}, {
"hash" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6120",
"triggerID" : "de94df1ece1dab7a903098a56a44eb305f24b801",
"triggerType" : "PUSH"
}, {
"hash" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6174",
"triggerID" : "02959b1adf82b4b8660965ac9b11101871ca8583",
"triggerType" : "PUSH"
}, {
"hash" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6178",
"triggerID" : "4bf310ccf064c7c5b00bebcb2aba4aa01ff1df78",
"triggerType" : "PUSH"
}, {
"hash" : "59444a040c40c4e03a5af12049a5000c049773f0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6199",
"triggerID" : "59444a040c40c4e03a5af12049a5000c049773f0",
"triggerType" : "PUSH"
}, {
"hash" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6222",
"triggerID" : "9f0ae0a1fc83618d08e9278394253f18af58c176",
"triggerType" : "PUSH"
}, {
"hash" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6224",
"triggerID" : "efce133ff21bb9116d1ad0330e404f150d0f7f41",
"triggerType" : "PUSH"
}, {
"hash" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6253",
"triggerID" : "456214637aca7d278a8536c381a5328d07c7f7cc",
"triggerType" : "PUSH"
}, {
"hash" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6276",
"triggerID" : "2bc18e5eaa267a051950bb50653f53e7d0e7d6b0",
"triggerType" : "PUSH"
}, {
"hash" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279",
"triggerID" : "1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7",
"triggerType" : "PUSH"
}, {
"hash" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec1eeba10ebeb04f76700ef19284b20480dd6e43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b49d3aa89486eea70ceb44d0aaa2e05d939a7c7 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6279)
* ec1eeba10ebeb04f76700ef19284b20480dd6e43 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8286] Standardize compaction and log compaction to use FileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13411:
URL: https://github.com/apache/hudi/pull/13411#discussion_r2151981377
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecord.java:
##########
@@ -168,6 +182,8 @@ public boolean isDelete(Schema recordSchema, Properties props) throws IOExceptio
}
if (this.data instanceof BaseAvroPayload) {
return ((BaseAvroPayload) this.data).isDeleted(recordSchema, props);
+ } else if (this.data instanceof HoodieMetadataPayload) {
Review Comment:
I think we can unify the `#isDelete` interface while we made a deep refactoring for `HoodieMetadataPayload` in the future, the current change is not a blocker and looks good to me, maybe we fire a JIRA task first to mark a TODO there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org