You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Vino1016 (via GitHub)" <gi...@apache.org> on 2026/02/02 10:40:39 UTC
[I] lots of Performance Warnings: DynamicRecord.schema Not Reused(even if i used a test-never-changed-schema) [iceberg]
Vino1016 opened a new issue, #15215:
URL: https://github.com/apache/iceberg/issues/15215
### Query engine
engine: Flink 2.0.0
iceberg: 1.10.0
### Question
Description:
Hi, Iām encountering a performance warning in Iceberg:
"Performance degraded as records with different schema are generated for the same table. Likely the DynamicRecord.schema is not reused. Reuse the same instance if the record schema is the same to improve performance."
This warning occurs for every data record.
My Code Logic:
I process upstream data in a process function. The schema for the test data is constant and does not change. Within this process, I construct a DynamicRecord for each day of data.
// Data processing (simplified)
DynamicRecord dynamicRecord = new DynamicRecord(
tableIdentifier,
ICEBERG_BRANCH,
this.newestSchema,
genericRowData,
this.newestPartitionSpec,
DistributionMode.NONE,
1
);
dynamicRecord.setEqualityFields(xxxx.getEqualityFieldColumns());
dynamicRecord.setUpsertMode(true);
out.collect(dynamicRecord);
The tableIdentifier, newestSchema, and newestPartitionSpec are initialized once in the open method and remain unchanged afterward.
Sink Configuration:
DynamicIcebergSink.forInput(dynamicRecordSingleOutputStreamOperator)
.cacheMaxSize(10)
.cacheRefreshMs(600000L)
.inputSchemasPerTableCacheMaxSize(10)
.immediateTableUpdate(true)
.overwrite(false)
.setAll(getSinkConfig(writeFormat))
.catalogLoader(TestS3Sink.getCatalogLoader(xxxx))
.generator(new DefaultDynamicRecordGenerator())
.append()
.setParallelism(1);
Custom Generator:
public class DefaultDynamicRecordGenerator implements DynamicRecordGenerator<DynamicRecord> {
@Override
public void open(OpenContext openContext) throws Exception {
DynamicRecordGenerator.super.open(openContext);
}
@Override
public void generate(DynamicRecord dynamicRecord, Collector<DynamicRecord> collector) throws Exception {
collector.collect(dynamicRecord);
}
}
Expected Behavior:
Since the schema does not change and remains the same object, Iceberg should reuse the same DynamicRecord.schema instance, rather than treating it as a new schema, which causes the warning.
Question:
Even though we are reusing the same schema instance, could this warning still appear due to some internal handling by Iceberg?
Apart from the way the schema is passed, are there any other factors that might cause Iceberg to treat it as a different schema, thus triggering this warning?
I would appreciate any help or suggestions. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Lots of Performance Warnings: DynamicRecord.schema Not Reused(even if i used a test-never-changed-schema) [iceberg]
Posted by "Vino1016 (via GitHub)" <gi...@apache.org>.
Vino1016 commented on issue #15215:
URL: https://github.com/apache/iceberg/issues/15215#issuecomment-3882477610
I found the root reason:
Root Cause Analysis
Flink Serialization Causes Schema Object Reference Invalidation
Upstream Operator: Although reusing the same Schema instance
Serialization Transfer: DynamicRecord is serialized and sent to downstream Writer
Downstream Writer: Each parallel instance gets a new Schema object after deserialization
Cache Invalidation: Iceberg's TableMetadataCache uses object references as keys, resulting in cache misses
Solution
Cache Schema objects in DynamicRecordGenerator:
JAVA
// Each Writer instance maintains its own Schema cache
private transient Schema cachedSchema;
// All records reuse the same object reference after compare
if (!icebergSchema.sameSchema(dynamicRecord.schema()))
dynamicRecord.setSchema(cachedSchema);
Results
ā
Performance warnings eliminated
ā
Cache hit rate 100%
ā
Supports Schema evolution detection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Lots of Performance Warnings: DynamicRecord.schema Not Reused(even if i used a test-never-changed-schema) [iceberg]
Posted by "Vino1016 (via GitHub)" <gi...@apache.org>.
Vino1016 closed issue #15215: Lots of Performance Warnings: DynamicRecord.schema Not Reused(even if i used a test-never-changed-schema)
URL: https://github.com/apache/iceberg/issues/15215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org