You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by "wenshao (via GitHub)" <gi...@apache.org> on 2026/08/01 20:25:58 UTC
Re: [PR] feat(format): schema evolution for the Java row codec [fory]
wenshao commented on code in PR #3714:
URL: https://github.com/apache/fory/pull/3714#discussion_r3696559577
##########
java/fory-format/src/main/java/org/apache/fory/format/type/TypeInference.java:
##########
@@ -258,6 +263,16 @@ private static Field inferField(String name, TypeRef<?> typeRef, TypeResolutionC
}
}
+ /**
+ * Infer a single named field from its Java type, used by schema-evolution code paths that need to
+ * reconstruct historical fields by name and type without going through a Java member.
+ */
+ static Field inferNamedField(String name, TypeRef<?> typeRef) {
+ TypeResolutionContext ctx =
+ new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true);
+ return inferField(name, typeRef, ctx);
Review Comment:
**[Critical]** `inferNamedField` creates a `TypeResolutionContext` with an empty walked-type-path, so `getEnclosingType()` returns `Object.class`. This misses bean-scoped custom codecs that the normal `inferSchema` path and `SchemaHistory.Wrapper.classify` both find via the actual bean class. — Failure scenario: register a custom codec scoped to `(Foo.class, Bar.class)` whose `encodedType()` returns `Baz`, and enable schema evolution on `Foo`. `SchemaHistory.enumerate` reconstructs fields via `inferNamedField` which calls `findCodec(Object.class, Bar)` and misses the `(Foo, Bar)` registration. The history-derived schema has `Bar`'s struct, but the generated codec expects `Baz`'s layout — if field counts differ, `fieldOfSchema` throws `IndexOutOfBoundsException`; if types differ, the writer silently corrupts the row.
```suggestion
static Field inferNamedField(String name, TypeRef<?> typeRef, Class<?> enclosingType) {
TypeResolutionContext ctx =
new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true)
.appendTypePath(enclosingType);
return inferField(name, typeRef, ctx);
```
_— qwen3-coder-plus via Qwen Code /review (v0.21.2)_
##########
java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java:
##########
@@ -0,0 +1,936 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.fory.format.type;
+
+import java.lang.reflect.AnnotatedElement;
+import java.lang.reflect.Type;
+import java.lang.reflect.TypeVariable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.function.UnaryOperator;
+import org.apache.fory.annotation.Internal;
+import org.apache.fory.collection.Tuple2;
+import org.apache.fory.format.annotation.ForySchema;
+import org.apache.fory.format.annotation.ForyVersion;
+import org.apache.fory.format.encoder.CustomCodec;
+import org.apache.fory.reflect.TypeRef;
+import org.apache.fory.type.Descriptor;
+import org.apache.fory.type.TypeResolutionContext;
+import org.apache.fory.type.TypeUtils;
+import org.apache.fory.util.StringUtils;
+
+/**
+ * Resolves the version history of a row-codec bean. Each entry exposes the schema as it appeared at
+ * a particular version, along with a strict hash that uniquely identifies the historical layout.
+ * Only used when {@code withSchemaEvolution()} is configured on the codec builder.
+ *
+ * <p>The hash mixes field names and nullability in addition to types, so that two schemas that
+ * differ only in field order or naming are distinguishable. This is intentionally a different hash
+ * from {@link DataTypes#computeSchemaHash} and is used only by versioning code paths.
+ */
+@Internal
+public final class SchemaHistory {
+
+ /** Implicit version of a live field that carries no {@link ForyVersion}. */
+ private static final int FIRST_VERSION = 1;
+
+ /** FNV-1a 64-bit offset basis, the seed for every strict-hash mix in this class. */
+ private static final long FNV_OFFSET_BASIS = 1469598103934665603L;
Review Comment:
**[Suggestion]** The FNV-1a 64-bit offset basis is `14695981039346656037` (`0xcbf29ce484222325`); the literal here is missing the trailing `7`, producing a different seed. Every other Fory runtime uses the correct value — C# `TypeResolver.cs:373`, C++ `type_index.h:53`, Rust `type_meta.rs:569`. — Concrete cost: the hash is self-consistent within Java so dispatch works today, but if schema-evolution hash comparison is ever extended cross-runtime or a user compares hash values for debugging, Java hashes will silently disagree. The comment explicitly claims this IS the FNV-1a 64-bit offset basis, making the code misleading.
```suggestion
private static final long FNV_OFFSET_BASIS = 0xcbf29ce484222325L;
```
_— qwen3-coder-plus via Qwen Code /review (v0.21.2)_
##########
java/fory-format/src/test/java/org/apache/fory/format/encoder/ImplementInterfaceTest.java:
##########
@@ -329,6 +331,41 @@ public void testListTooLazy() {
Assert.assertEquals(deserializedBean.f1().get(0).f1(), 42);
}
+ public interface MapOuter {
+ Map<String, ListInner> f1();
+ }
+
+ static class MapOuterImpl implements MapOuter {
+ private final Map<String, ListInner> f1;
+
+ MapOuterImpl(final Map<String, ListInner> f1) {
+ this.f1 = f1;
+ }
+
+ @Override
+ public Map<String, ListInner> f1() {
+ return f1;
+ }
+ }
+
+ /**
+ * Interface bean as a map value. Type inference reaches the map value type through {@code
+ * isSupported}, which must recognize the interface as a synthesizable bean the same way it does
+ * for a direct field or list element.
+ */
+ @Test
+ public void testMapValueInterface() {
+ final Map<String, ListInner> map = new HashMap<>();
Review Comment:
**[Suggestion]** The `TypeUtils.isSupported` ctx-threading fix is symmetric across map key and value, but only the value side is exercised by a test; no test uses an interface bean as a map key through this path. — Concrete cost: a future refactoring that accidentally reverts only the key-side call to single-arg `isSupported` would cause interface beans used as map keys to be rejected. No existing test would turn red: `testMapValueInterface` uses `String` as the key (always supported), and the map-key evolution tests in `SchemaEvolutionStressTest` use concrete `@Data` classes.
Suggested fix: add a counterpart `testMapKeyInterface` that uses an interface bean as the map key.
_— qwen3-coder-plus via Qwen Code /review (v0.21.2)_
##########
java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java:
##########
@@ -55,23 +66,246 @@ public class MapCodecBuilder<M extends Map<?, ?>> extends BaseCodecBuilder<MapCo
public Supplier<MapEncoder<M>> build() {
loadMapInnerCodecs();
- final var mapEncoderFactory = generatedMapEncoder();
+ final Class<?> valClass = schemaEvolution ? evolutionBean() : null;
+ final Class<?> keyClass = schemaEvolution ? keyEvolutionBean() : null;
+ if (valClass == null && keyClass == null) {
+ final var mapEncoderFactory = generatedMapEncoder();
+ return new Supplier<MapEncoder<M>>() {
+ @Override
+ public MapEncoder<M> get() {
+ final BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField);
+ final BinaryArrayWriter valWriter =
+ codecFormat.newArrayWriter(valField, keyWriter.getBuffer());
+ final var codec = mapEncoderFactory.apply(keyWriter, valWriter);
+ return new BufferResettingMapEncoder<>(
+ initialBufferSize,
+ keyWriter,
+ valWriter,
+ new BinaryMapEncoder<M>(
+ codecFormat, field, valWriter, keyWriter, codec, sizeEmbedded));
+ }
+ };
+ }
+ return buildVersioned(valClass, keyClass);
+ }
+
+ /**
+ * Whether the value position takes the evolution path, and a representative bean for naming. A
+ * directly-typed bean (versioned or not) takes the path so the strict-hash prefix is always
+ * present and an evolution-on consumer can detect a flag-mismatched producer cleanly; a bean
+ * nested inside a list/map/array value is found by descending the wrapper. Null when the value
+ * carries no bean. The per-version enumeration over every reachable bean in the value position is
+ * done by {@link #buildElementSchemaHistory}.
+ */
+ private Class<?> evolutionBean() {
+ return SchemaHistory.evolutionBean(valType, typeCtx());
+ }
+
+ /**
+ * Bean this map's key evolves on, reachable through the key type, mirroring {@link
+ * #evolutionBean()} for the value. Null when the key carries no bean. A versioned key is read at
+ * the matching historical layout selected by the map header's combined hash, so an evolving key
+ * no longer corrupts silently.
+ */
+ private Class<?> keyEvolutionBean() {
+ return SchemaHistory.evolutionBean(keyType, typeCtx());
+ }
+
+ /**
+ * Versions for one map position: the history's entries, or a single null when it does not evolve.
+ */
+ private static List<SchemaHistory.VersionedSchema> positionVersions(SchemaHistory history) {
+ return history == null ? Collections.singletonList(null) : history.versions();
+ }
+
+ /** Strict hash contributed by one position: its schema's hash, or the non-bean constant. */
+ private static long positionHash(SchemaHistory.VersionedSchema vs) {
+ return vs == null ? NON_BEAN_POSITION_HASH : vs.strictHash();
+ }
+
+ private Supplier<MapEncoder<M>> buildVersioned(final Class<?> valClass, final Class<?> keyClass) {
+ // The map's own key and value are independent positions on the wire (each its own array), and
+ // the map header's combined hash identifies the (key-layout, value-layout) pair. Each position
+ // is enumerated over every versioned bean reachable through its wrappers, so a position that
+ // itself wraps more than one bean (such as a value typed Map<KBean, VBean>) evolves all of
+ // them.
+ // A position with no bean contributes a single current-only layout, so the cross-product
+ // degenerates to the evolving position's versions (or to just the current layout when neither
+ // evolves, i.e. a non-versioned bean that still needs the hash prefix for flag-mismatch
+ // detection).
+ SchemaHistory valHistory =
+ valClass == null ? null : buildElementSchemaHistory(field.name(), valType);
+ SchemaHistory keyHistory =
+ keyClass == null ? null : buildElementSchemaHistory(field.name(), keyType);
+ SchemaHistory.VersionedSchema valCurrent = valHistory == null ? null : valHistory.current();
+ SchemaHistory.VersionedSchema keyCurrent = keyHistory == null ? null : keyHistory.current();
+
+ // Index one deferred projection source per (key-version, value-version) combination, keyed by
+ // the combined hash. The current/current combination is the hot path and is handled by the
+ // unsuffixed current codec, so it is skipped here. A null history means that position does not
+ // evolve and contributes only its current entry. Building the index compiles nothing: a
+ // combination's codec classes are generated the first time its hash is decoded. The collision
+ // guards below still run eagerly over the full cross-product, so a hash clash fails fast at
+ // build rather than surfacing on an unlucky decode.
+ //
+ // The full cross-product is required even when key and value are the same bean family: a writer
+ // can pin them to different versions via distinct type arguments (Map<DefaultsV1, DefaultsV2>),
+ // so off-diagonal pairs are reachable. See evolveMapSameBeanKeyAndValueCrossCombos.
+ List<SchemaHistory.VersionedSchema> valVersions = positionVersions(valHistory);
+ List<SchemaHistory.VersionedSchema> keyVersions = positionVersions(keyHistory);
+ LongMap<BinaryMapEncoder.ProjectionSource> projectionSources = new LongMap<>();
+ for (SchemaHistory.VersionedSchema valVs : valVersions) {
+ for (SchemaHistory.VersionedSchema keyVs : keyVersions) {
+ if (valVs == valCurrent && keyVs == keyCurrent) {
+ continue;
+ }
+ // The map header carries a single hash, so combine the key and value strict hashes into
+ // one 64-bit map-layout hash that identifies the (key-version, value-version) combination
+ // jointly. The collision check below proves these combined hashes are unique at build time.
+ long hash = SchemaHistory.combineHashes(positionHash(keyVs), positionHash(valVs));
+ if (projectionSources.containsKey(hash)) {
+ throw new IllegalStateException(
+ "Combined (key, value) schema-hash collision for map "
+ + mapType
+ + ": two distinct version combinations produced the same map-layout hash. "
+ + "Please file an issue with the key and value bean definitions.");
+ }
+ projectionSources.put(
+ hash, new ProjectionSource(valClass, keyClass, valVs, keyVs, valCurrent, keyCurrent));
+ }
+ }
+ final var currentFactory = generatedMapEncoder();
+ long currentHash =
+ SchemaHistory.combineHashes(positionHash(keyCurrent), positionHash(valCurrent));
+ // The decode hot path matches currentHash before consulting the projection map, so a projection
+ // colliding with it would be shadowed and never dispatched to. Prove that cannot happen.
+ if (projectionSources.containsKey(currentHash)) {
+ throw new IllegalStateException(
+ "Combined (key, value) schema-hash collision for map "
+ + mapType
+ + ": a historical version combination produced the same map-layout hash as the "
+ + "current schema. Please file an issue with the key and value bean definitions.");
+ }
return new Supplier<MapEncoder<M>>() {
@Override
public MapEncoder<M> get() {
- final BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField);
- final BinaryArrayWriter valWriter =
- codecFormat.newArrayWriter(valField, keyWriter.getBuffer());
- final var codec = mapEncoderFactory.apply(keyWriter, valWriter);
+ BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField);
+ BinaryArrayWriter valWriter = codecFormat.newArrayWriter(valField, keyWriter.getBuffer());
+ var codec = currentFactory.apply(keyWriter, valWriter);
return new BufferResettingMapEncoder<>(
initialBufferSize,
keyWriter,
valWriter,
- new BinaryMapEncoder<M>(codecFormat, field, valWriter, keyWriter, codec, sizeEmbedded));
+ new BinaryMapEncoder<M>(
+ codecFormat,
+ field,
+ valWriter,
+ keyWriter,
+ codec,
+ sizeEmbedded,
+ currentHash,
+ projectionSources,
+ fory));
}
};
}
+ /**
+ * Generate the projection row codec(s) and projection map codec for one (key-version,
+ * value-version) combination, and rebuild the map field with each position projected onto its
+ * historical schema. A position at its current schema gets an empty suffix and keeps its current
+ * field, so a map where only one side evolves pays no codegen for the unchanged side.
+ */
Review Comment:
**[Suggestion]** Orphaned Javadoc block: this comment is immediately followed by a second Javadoc block (the real documentation for `projectedPositionField`), so it attaches to no declaration and is invisible to javadoc and IDEs. — Concrete cost: the text describes `ProjectionSource.compile`'s contract but sits before `projectedPositionField`. A maintainer reading the source sees what looks like the method's contract, but it documents the wrong method; the real `compile` method has no equivalent top-level summary. The stale block will drift out of sync because nothing tooling-wise references it.
Suggested fix: move this Javadoc block to precede the `ProjectionSource.compile` method, or delete it if `compile`'s existing inline comments are sufficient.
_— qwen3-coder-plus via Qwen Code /review (v0.21.2)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org