You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "avi-sanwal (via GitHub)" <gi...@apache.org> on 2026/05/14 17:13:45 UTC
[PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
avi-sanwal opened a new pull request, #28161:
URL: https://github.com/apache/flink/pull/28161
## What is the purpose of the change
This pull request fixes GenericRecord Avro state migration when a compatible schema evolution adds fields with defaults.
During state migration, Flink can deserialize old state bytes with the previous GenericRecord schema and then serialize that value with the new serializer. The migrated value is still shaped like the old schema, so writing it directly with the new writer schema can fail in Avro's GenericDatumWriter before Avro schema resolution applies defaults.
## Brief change log
- Resolve GenericRecord values whose embedded schema differs from the serializer runtime schema before writing them.
- Keep the existing same-schema serialization path unchanged.
- Add a serializer snapshot regression test for old-schema read followed by new-schema write.
## Verifying this change
This change added tests and can be verified as follows:
- Added `AvroSerializerSnapshotTest#migratedGenericRecordShouldBeSerializedWithNewSchema`.
- Ran `./mvnw -pl flink-formats/flink-avro spotless:apply`.
- Ran `./mvnw -pl flink-formats/flink-avro -Dtest=AvroSerializerSnapshotTest#migratedGenericRecordShouldBeSerializedWithNewSchema -Dcheckstyle.skip -Drat.skip -Dspotless.check.skip test`; this did not reach the test because the local Maven repository is missing required `2.4-SNAPSHOT` Flink artifacts.
- Ran `./mvnw -pl flink-formats/flink-avro -am -Dtest=AvroSerializerSnapshotTest#migratedGenericRecordShouldBeSerializedWithNewSchema -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dcheckstyle.skip -Drat.skip -Dspotless.check.skip test`; this failed before `flink-avro` in `flink-table-planner` Scala compilation due missing table runtime type classes in this checkout.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: yes
- The runtime per-record code paths (performance sensitive): yes
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes, state restore/migration
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes: OpenAI Codex GPT-5
Generated-by: OpenAI Codex GPT-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on code in PR #28161:
URL: https://github.com/apache/flink/pull/28161#discussion_r3263403085
##########
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java:
##########
@@ -325,6 +330,30 @@ private void initializeAvro() {
this.avroData = factory.getAvroData();
}
+ private T resolveGenericRecord(T value) throws IOException {
Review Comment:
Thanks, addressed in 85bd596278e. I added a short method comment clarifying that this handles GenericRecord values restored during state migration that still carry the previous schema, before the writer indexes fields by position.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5290854187
@davidradl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5277547053
Any interested review? Would love to see this PR merge soon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28161:
URL: https://github.com/apache/flink/pull/28161#discussion_r3257810709
##########
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java:
##########
@@ -325,6 +330,30 @@ private void initializeAvro() {
this.avroData = factory.getAvroData();
}
+ private T resolveGenericRecord(T value) throws IOException {
Review Comment:
It would be useful to include a comment as to the intent of this new method. I assume it is when the generic record does not match the type we think it is , due to schema migrtaion.
##########
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java:
##########
@@ -325,6 +330,30 @@ private void initializeAvro() {
this.avroData = factory.getAvroData();
}
+ private T resolveGenericRecord(T value) throws IOException {
Review Comment:
It would be useful to include a comment as to the intent of this new method. I assume it is when the generic record does not match the type we think it is , due to schema migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5290848436
Thanks for taking a look and for spelling these out. I pushed a commit with the diagnostic and negative-test follow-ups.
> 1. Per-record allocation in the hot path (performance)
The extra schema-resolution work only runs when the GenericRecord schema differs from the serializer runtime schema; matching schemas continue through the existing path unchanged. I also deliberated caching the migration writer/reader, but caching the writer would not remove the field traversal or conversion work. I left that out to avoid adding retention and complexity to a migration-only path.
> 2. No IOException wrapping / error message
Added wrapping that preserves the original cause and includes both the writer and reader schemas in the error message.
> 3. Objects.equals on Avro Schema
I checked Avro Schema.equals(): it already performs an identity check before structural comparison, so an explicit recordSchema == runtimeSchema || ... check would be redundant.
> 4. Test helper methods serialize() / deserialize() — already in the test class?
Confirmed—the existing helpers exercise AvroSerializer.serialize() and deserialize(). The test follows the relevant migration sequence: deserialize with the previous serializer, then serialize with the current one.
> 5. No coverage for incompatible schema (negative test)
Added a negative test for an old record that is missing a required field in the reader schema. It verifies the contextual IOException and preserves the underlying AvroTypeException as the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-4452996536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8734c111fafc96867a2ec3774f4cf83ab829485f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8734c111fafc96867a2ec3774f4cf83ab829485f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8734c111fafc96867a2ec3774f4cf83ab829485f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5406451961
@davidradl , gentle reminder - please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5580165440
@davidradl , please unblock the PR if you have no concerns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub) via issues" <is...@flink.apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-6560688354
@SEPURI-SAI-KRISHNA and @davidradl - please approve so that this PR can be merged if there are no major pending issues. Any minor issues can be then taken up in future PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-4941556622
@davidradl - please have a second look and if things look good, approve this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on code in PR #28161:
URL: https://github.com/apache/flink/pull/28161#discussion_r3263403688
##########
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java:
##########
@@ -325,6 +330,30 @@ private void initializeAvro() {
this.avroData = factory.getAvroData();
}
+ private T resolveGenericRecord(T value) throws IOException {
+ if (value == null || !isGenericRecord(type)) {
+ return value;
+ }
+
+ final GenericRecord record = (GenericRecord) value;
+ final Schema recordSchema = record.getSchema();
+ if (Objects.equals(recordSchema, runtimeSchema)) {
+ return value;
Review Comment:
Thanks, addressed in 85bd596278e. I added coverage for the same-schema GenericRecord path, and also added a nullable-union GenericRecord roundtrip to exercise the null return branch through public serializer behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28161:
URL: https://github.com/apache/flink/pull/28161#discussion_r3257818750
##########
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java:
##########
@@ -325,6 +330,30 @@ private void initializeAvro() {
this.avroData = factory.getAvroData();
}
+ private T resolveGenericRecord(T value) throws IOException {
+ if (value == null || !isGenericRecord(type)) {
+ return value;
+ }
+
+ final GenericRecord record = (GenericRecord) value;
+ final Schema recordSchema = record.getSchema();
+ if (Objects.equals(recordSchema, runtimeSchema)) {
+ return value;
Review Comment:
can we add a test to test this return?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "avi-sanwal (via GitHub)" <gi...@apache.org>.
avi-sanwal commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5934533159
Thanks for the concrete reproduction and the heap state trace, @SEPURI-SAI-KRISHNA. `copy()` was using the runtime schema to index a record that still carried the previous schema.
I pushed a commit that resolves a mismatched `GenericRecord` before the deep copy. The `copy(from, reuse)` overload delegates to the same path. I added a regression test for your `A_C_D` to `A_B_C_D` example, plus an incompatible-schema test that checks the contextual exception and preserved cause. The focused Avro serializer tests and Spotless check pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38023][formats] Fix GenericRecord Avro state migration [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #28161:
URL: https://github.com/apache/flink/pull/28161#issuecomment-5581136749
`copy()` has the same problem and isn't in the diff. It reaches fields by position too, via `avroData.deepCopy(runtimeSchema, from)`, so a record still carrying the previous schema fails there the way it did in `serialize()`. On master, with the schema pairs your new tests define:
```java
AvroSerializer<GenericRecord> s = new AvroSerializer<>(GenericRecord.class, A_B_C_D);
GenericRecord old =
new GenericRecordBuilder(A_C_D).set("a", "A").set("c", "C").set("d", "D").build();
s.copy(old); // ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3
```
FIRST_NAME against FIRST_REQUIRED_LAST_OPTIONAL fails the same way. Nothing covers it today: `SerializerTestBase#testCopy` only copies records the same serializer produced.
It is reachable on HashMapStateBackend. `HeapKeyedStateBackend` leaves non-TTL migration until checkpoint time, so restored values sit in the state table with the old schema while the new serializer is installed, and `CopyOnWriteStateMap` calls `getStateSerializer().copy(e.state)` on copy-on-write.
`TypeSerializer#copy` doesn't declare `IOException`, so routing it through the same helper would need a `FlinkRuntimeException` wrap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org