You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "harshagudladona (via GitHub)" <gi...@apache.org> on 2025/10/09 15:28:04 UTC
[I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
harshagudladona opened a new issue, #14070:
URL: https://github.com/apache/hudi/issues/14070
### Bug Description
**Environment:**
Spark 3.5
Java 11
Hudi 0.14 and 1.0.2
Storage S3
**What happened:**
We are noticing about 20-30% write performance degradation between Hudi 0.14 and 1.0.2
We have profiled the executors by writing the same data between executors running both versions, and here is the first of few suspects. We will open a separate bug report as we find more suspects.
Small file Handling:
In both versions, the HoodieMergeHelper.runMerge function, the following condition ends up being false, without any change to the writer schema during the test
```
// Check whether the writer schema is simply a projection of the file's one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
boolean isPureProjection = schemaEvolutionTransformerOpt.isEmpty()
&& isStrictProjectionOf(readerSchema, writerSchema);
```
Due to this the record is always rewritten with function rewriteRecordWithNewSchema
```
executor = ExecutorFactory.create(writeConfig, recordIterator, new UpdateHandler(mergeHandle), record -> {
HoodieRecord newRecord;
if (schemaEvolutionTransformerOpt.isPresent()) {
newRecord = schemaEvolutionTransformerOpt.get().apply(record);
} else if (shouldRewriteInWriterSchema) {
newRecord = record.rewriteRecordWithNewSchema(recordSchema, writeConfig.getProps(), writerSchema);
} else {
newRecord = record;
}
```
In this function in hudi 1.x in HoodieAvroUtils.rewriteRecordWithNewSchema, the following check is added, which seems to be taking significant CPU time.
```
if (oldAvroSchema.equals(newSchema)) {
// there is no need to rewrite if the schema equals.
return oldRecord;
}
//
```
1.x flamegraph
<img width="2325" height="526" alt="Image" src="https://github.com/user-attachments/assets/4e934182-b445-40dc-be10-40e90bb9af30" />
0.14 flamegraph
<img width="2326" height="587" alt="Image" src="https://github.com/user-attachments/assets/fc132bef-99e1-4330-8fe6-6928cd48173d" />
hostspot in 1.0.2
<img width="2332" height="642" alt="Image" src="https://github.com/user-attachments/assets/98e63658-aabb-4ad9-819b-cc05703eecc5" />
**What you expected:**
No performance degradation
**Steps to reproduce:**
1. Write any dataset in 0.14 and 1.0.2 and observe the difference.
### Environment
**Hudi version: 0.14 and 1.0.2**
**Query engine: Spark** (Spark/Flink/Trino etc)
**Relevant configs: Same configurations as the one reported here: https://github.com/apache/hudi/issues/13995**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "harshagudladona (via GitHub)" <gi...@apache.org>.
harshagudladona commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3407168975
We cannot share the whole schema but here is a snippet there the reader and the writer schema will differ
WRITER:
```
{
"name" : "enrichments",
"type" : [ "null", {
"type" : "record",
"name" : "Enrichments",
"namespace" : "com.secureworks.platform.schemas.core.common.Types",
"fields" : [ {
"name" : "attack_technique_ids",
"type" : [ "null", {
"type" : "array",
"items" : [ "null", "string" ]
} ],
"default" : null
}, {
"name" : "rule_id_to_techniques",
"type" : [ "null", {
"type" : "array",
"items" : {
"type" : "record",
"name" : "KeyAndValues",
"fields" : [ {
"name" : "key",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "values",
"type" : [ "null", {
"type" : "array",
"items" : [ "null", "string" ]
} ],
"default" : null
} ]
}
} ],
"default" : null
} ]
} ],
"default" : null
}
```
READER:
```
{
"name" : "enrichments",
"type" : [ "null", {
"type" : "record",
"name" : "enrichments",
"namespace" : "",
"fields" : [ {
"name" : "attack_technique_ids",
"type" : [ "null", {
"type" : "array",
"items" : "string"
} ],
"default" : null
}, {
"name" : "rule_id_to_techniques",
"type" : [ "null", {
"type" : "array",
"items" : {
"type" : "record",
"name" : "array",
"fields" : [ {
"name" : "key",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "values",
"type" : [ "null", {
"type" : "array",
"items" : "string"
} ],
"default" : null
} ]
}
} ],
"default" : null
} ]
} ],
"default" : null
}
```
Notice the difference in the ` "name" : "KeyAndValues",` in writer and `"name" : "array",` in reader. The reason for this is that the reader is schema derived from a basefile is a conversion of the parquet schema (MessageType) into an avro schema. In that conversion to/from avro(Schema) to parquet(MessageType) there are differences like this, that will never match. See the snippet below from org.apache.parquet.avro.AvroSchemaConverter
```
} else if (type.equals(Schema.Type.ARRAY)) {
if (writeOldListStructure) {
return ConversionPatterns.listType(repetition, fieldName,
convertField("array", schema.getElementType(), REPEATED, schemaPath));
} else {
return ConversionPatterns.listOfElements(repetition, fieldName,
convertField(AvroWriteSupport.LIST_ELEMENT_NAME, schema.getElementType(), schemaPath));
}
}
```
What I tried to address this is the following:
BEFORE:
```
@Override
public void runMerge(HoodieTable<?, ?, ?, ?> table,
HoodieMergeHandle<?, ?, ?, ?> mergeHandle) throws IOException {
HoodieWriteConfig writeConfig = table.getConfig();
HoodieBaseFile baseFile = mergeHandle.baseFileForMerge();
HoodieRecord.HoodieRecordType recordType = table.getConfig().getRecordMerger().getRecordType();
HoodieFileReader baseFileReader = HoodieIOFactory.getIOFactory(
table.getStorage().newInstance(mergeHandle.getOldFilePath(), table.getStorageConf().newInstance()))
.getReaderFactory(recordType)
.getFileReader(writeConfig, mergeHandle.getOldFilePath());
HoodieFileReader bootstrapFileReader = null;
Schema writerSchema = mergeHandle.getWriterSchemaWithMetaFields();
Schema readerSchema = baseFileReader.getSchema();
// In case Advanced Schema Evolution is enabled we might need to rewrite currently
// persisted records to adhere to an evolved schema
Option<Function<HoodieRecord, HoodieRecord>> schemaEvolutionTransformerOpt =
composeSchemaEvolutionTransformer(readerSchema, writerSchema, baseFile, writeConfig, table.getMetaClient());
// Check whether the writer schema is simply a projection of the file's one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
boolean isPureProjection = schemaEvolutionTransformerOpt.isEmpty()
&& isStrictProjectionOf(readerSchema, writerSchema);
...
```
AFTER:
```
@Override
public void runMerge(HoodieTable<?, ?, ?, ?> table,
HoodieMergeHandle<?, ?, ?, ?> mergeHandle) throws IOException {
HoodieWriteConfig writeConfig = table.getConfig();
HoodieBaseFile baseFile = mergeHandle.baseFileForMerge();
HoodieRecord.HoodieRecordType recordType = table.getConfig().getRecordMerger().getRecordType();
HoodieFileReader baseFileReader = HoodieIOFactory.getIOFactory(
table.getStorage().newInstance(mergeHandle.getOldFilePath(), table.getStorageConf().newInstance()))
.getReaderFactory(recordType)
.getFileReader(writeConfig, mergeHandle.getOldFilePath());
HoodieFileReader bootstrapFileReader = null;
Schema readerSchema = baseFileReader.getSchema();
Schema writerSchema = mergeHandle.getWriterSchemaWithMetaFields();
// This is necessary to ensure the writer schema is compatible with the parquet schema
// as the Avro schema used to write the parquet file might be different than the
// writer schema (eg: default values added for missing fields, documentation etc)
MessageType parquetSchema = new AvroSchemaConverter(table.getStorage().getConf().unwrapAs(Configuration.class)).convert(writerSchema);
Schema parquetCompatibleWriteSchema = new AvroSchemaConverter(table.getStorage().getConf().unwrapAs(Configuration.class)).convert(parquetSchema);
// In case Advanced Schema Evolution is enabled we might need to rewrite currently
// persisted records to adhere to an evolved schema
Option<Function<HoodieRecord, HoodieRecord>> schemaEvolutionTransformerOpt =
composeSchemaEvolutionTransformer(readerSchema, writerSchema, baseFile, writeConfig, table.getMetaClient());
// Check whether the writer schema is simply a projection of the file's one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
boolean isStrictProjection = isStrictProjectionOf(readerSchema, parquetCompatibleWriteSchema);
boolean isPureProjection = schemaEvolutionTransformerOpt.isEmpty()
&& isStrictProjection;
```
I used the same converter to make sure the writer schema is transformed into parquet Parquet-friendly version. This fixed the problem of false-positive failure of `isStrictProjectionOf` thereby preventing a record rewrite.
Additionally, In the HoodieMergeHandle
```
protected void writeToFile(HoodieKey key, HoodieRecord<T> record, Schema schema, Properties prop, boolean shouldPreserveRecordMetadata) throws IOException {
// NOTE: `FILENAME_METADATA_FIELD` has to be rewritten to correctly point to the
// file holding this record even in cases when overall metadata is preserved
MetadataValues metadataValues = new MetadataValues().setFileName(newFilePath.getName());
HoodieRecord populatedRecord = record.prependMetaFields(schema, writeSchemaWithMetaFields, metadataValues, prop);
if (shouldPreserveRecordMetadata) {
fileWriter.write(key.getRecordKey(), populatedRecord, writeSchemaWithMetaFields);
} else {
fileWriter.writeWithMetadata(key, populatedRecord, writeSchemaWithMetaFields);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "harshagudladona (via GitHub)" <gi...@apache.org>.
harshagudladona commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3397384190
Sorry, I could not respond earlier, I will surely do. But I have a theory on why this is happening, which is related to schema verification. The reader schema is from the parquet schema and not the "parquet.avro.schema" in the metadata, which is the source schema. Any subtle differences in both the versions, such as defaults, doc, namespace will cause the isStrictProjectionOf to be false along with "oldAvroSchema.equals(newSchema)" check.
We are testing a version of the code where the source schema is transformed to be same as the parquet schema, which will result in the schema check to always result in true. We will get back with results in a day or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3409376829
> // This is necessary to ensure the writer schema is compatible with the parquet schema
// as the Avro schema used to write the parquet file might be different than the
// writer schema (eg: default values added for missing fields, documentation etc)
Configuration conf = table.getStorage().getConf().unwrapAs(Configuration.class);
conf.set(AvroSchemaConverter.ADD_LIST_ELEMENT_RECORDS, "false");
conf.set(ParquetInputFormat.STRICT_TYPE_CHECKING, "false");
MessageType parquetSchema = new AvroSchemaConverter(conf).convert(writerSchema);
Schema parquetCompatibleWriteSchema = new AvroSchemaConverter(conf).convert(parquetSchema);
The problem seems only exists for parquet format, but the code here is also used for other formats, like ORC, so we can not simply make such modification. I prefer raising a hot fix for the special type `enum` currently (here is the hotfix PR https://github.com/apache/hudi/pull/14094 for master branch), and the community is planing to refactor the type system and schema, the problem can be solved thoroughly then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3399615795
@harshagudladona are you using code with any customized modification? Actually we can not reproduce the problem locally. It would be great if you can provide the program to reproduce the problem, thks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3393426741
hey @harshagudladona : We tested with a wide schema (5000 col schema) and could not find any perf difference. Can you share more details on the schema being used. Was it having lot of nested fields? complex types etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3388023163
The second issue should be addressed by fix: https://github.com/apache/hudi/pull/12949, maybe we missed something, as for the 1st issue, needs some investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "harshagudladona (via GitHub)" <gi...@apache.org>.
harshagudladona commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3402918681
We have not done any modifications to the hudi codebase. However, we have implemented a schema provider that transforms proto to avro schema.
The use case is very straightforward. We have a simple COW/MOR table with INSERT and in-line small file handling, meaning append to one of the small files.
We think there is a computationally intensive schema equality check that is new in 1.0.2, which is slowing down the record writes in 1.0.2. Like I mentioned earlier, in HoodieMergeHelper the reader schema is derived from
`Schema readerSchema = baseFileReader.getSchema();`
This is a conversion of MessageType (parquet schema) to avro schema, that is along the lines of, say
`Schema avroFromParquetSchema = new AvroSchemaConverter(table.getStorage().getConf().unwrapAs(Configuration.class)).convert(parquetSchema);`
The method isStrictProjectionOf accepts this avroFromParquetSchema and the writer schema, which are rarely equal. This will cause the merge operation to almost always rewrite the record unnecessarily.
I cannot think of a way to provide reproducible way that simulates our setup. The test setup is exactly the same as the one described in this issue https://github.com/apache/hudi/issues/13995. We have run ingestion from the same source on the same data from 0.14 and 1.0.2, and we see that 1.0.2 takes longer to finish, and in each iteration of processing records, the write stage takes longer(20% longer) in 1.0.2.
With this being said, I am working on changing how the input schemas are provided to isStrictProjectionOf slightly to see if this can help avoid unnecessary rewrites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3404321025
> Any subtle differences in both the versions, such as defaults, doc, namespace will cause the isStrictProjectionOf to be false along with "oldAvroSchema.equals(newSchema)" check.
@harshagudladona AFAIK, doc in schema will not influence the result of `isStrictProjectionOf`. If it's not easy to provide the reproducible program, can you just past the `readerSchema` and `writeSchema` here, thks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write performance degradation between hudi 0.14 and 1.0.2 [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #14070:
URL: https://github.com/apache/hudi/issues/14070#issuecomment-3388117015
Can you share your table schema which is very related with this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org