You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/09/03 11:33:09 UTC
[PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
raminqaf opened a new pull request, #29087:
URL: https://github.com/apache/flink/pull/29087
## What is the purpose of the change
Second subtask of FLIP-604, building directly on FLINK-40486 (merged as #29041), which introduced the `UUID` logical type at the type-system layer only. After that change a `UUID` value still could not be materialized, serialized, or returned at runtime. This change wires `UUID` through the table runtime, code generation, and the DataStream type stack so `UUID`
columns work end to end.
Internally a `UUID` is stored as its canonical 16-byte big-endian encoding, reusing them `byte[]` representation and `getBinary`/`writeBinary` access already used by `BINARY`/`VARBINARY`. The external/default conversion class is `java.util.UUID`.
## Brief change log
- Internal representation and serialization: `LogicalTypeUtils.toInternalConversionClass` returns `byte[]`; `InternalSerializers`, `BinaryWriter`, `BinaryArrayWriter`, the interpreted `RowData`/`ArrayData` field getters, and `BinaryArrayData` sizing reuse the `BINARY`/`VARBINARY` path.
- Code generation: `CodeGenUtils` (type term, hashing, row read/write) and `GenerateUtils` literal handling emit the `byte[]` path for `UUID`; `RexLiteralUtil` converts a `java.util.UUID` literal to 16 big-endian bytes.
- External conversion: new `UuidUuidConverter` (`byte[]` <-> `java.util.UUID`), registered in `DataStructureConverters`.
- DataStream API: new `UuidSerializer` and `UuidComparator`, plus `Types.UUID`. The comparator uses unsigned big-endian byte order (the type's defined order), not the signed `java.util.UUID.compareTo`.
- Reflective extraction: `UUID` is auto-extracted for Table/SQL via `ClassDataTypeConverter` but intentionally not auto-registered for the DataStream API (`BasicTypeInfo.TYPES`); see below.
- Planner plumbing: `FlinkRelMdSize` (16-byte estimate) and `ExpressionReducer` (skip constant folding, as for `VARIANT`).
## Verifying this change
This change added tests and can be verified as follows:
- `UuidSerializerTest` / `UuidComparatorTest` (`flink-core`): serialization round-trip and unsigned ordering, including normalized-key consistency; the sorted test data spans the signed-long boundary (`0x7FFF...` vs `0x8000...`).
- `DataStructureConvertersTest`: `byte[]` <-> `java.util.UUID` round-trip.
- `ClassDataTypeConverterTest` / `DataTypeExtractorTest`: Table/SQL auto-extraction of a `java.util.UUID` field to `DataTypes.UUID()`.
- `UuidITCase` (`flink-table-planner`): end-to-end `SELECT`, `ARRAY[...]`, multi-row `VALUES`, `UNION`/`CASE`, `MAP` value, and nested `ROW`, collecting `java.util.UUID` back through the full plan/codegen/execution stack.
## Design notes / trade-offs for reviewers
- **DataStream extraction is opt-in.** Registering `java.util.UUID` for automatic reflective extraction in the DataStream API would silently switch the serializer of existing `java.util.UUID` fields from Kryo to `UuidSerializer`, breaking savepoint compatibility on upgrade. DataStream users therefore opt in via `Types.UUID`; automatic extraction is planned for the next major version. Table/SQL does auto-extract, since a plain `UUID` field there otherwise requires an explicit RAW annotation, so there is no silent change.
- **Comparator order** is unsigned big-endian to match the type's canonical order and the SQL side, deliberately differing from `java.util.UUID.compareTo` (signed per half).
## Related tickets (FLIP-604)
- Depends on: FLINK-40486 — Introduce the UUID logical type (merged, #29041).
- Follow-ups, handled in separate subtasks:
- FLINK-40487 — casts `UUID <-> STRING` / `BINARY(16)` (note: `.print()` depends on the
to-string cast).
- FLINK-40488 — comparison and ordering in SQL (`ORDER BY` / `GROUP BY` / joins);
`TypeCheckUtils.isComparable` still excludes `UUID` in this PR.
- FLINK-40489 — `UUID_V4` / `UUID_V7` functions.
- FLINK-40494 — documentation.
- Not addressed here: compiled-plan `RexNode` literal serde, left as a follow-up as it was for `VARIANT`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes — adds
`Types.UUID` and `BasicTypeInfo.UUID_TYPE_INFO`
- The serializers: yes — new `UuidSerializer`; `UUID` internal state uses the byte-array serializer
- The runtime per-record code paths (performance sensitive): yes — per-record field getters/
writers and generated field access for `UUID`
- Anything that affects deployment or recovery: JobManager (and its components),
Checkpointing, Kubernetes/Yarn, ZooKeeper: no (new type; no state migration for existing jobs)
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes (as part of FLIP-604)
- If yes, how is the feature documented? not documented in this PR; user-facing docs are
handled in a separate FLIP-604 subtask (FLINK-40494)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925479631
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/BasicTypeInfo.java:
##########
@@ -338,5 +344,9 @@ private static <X> TypeComparator<X> instantiateComparator(
TYPES.put(BigInteger.class, BIG_INT_TYPE_INFO);
TYPES.put(BigDecimal.class, BIG_DEC_TYPE_INFO);
TYPES.put(Instant.class, INSTANT_TYPE_INFO);
+ // UUID is intentionally not registered here. Automatic reflective extraction of
+ // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+ // avoid silently changing the serializer of existing java.util.UUID fields (currently
+ // handled by Kryo). This is planned to change in the next major Flink version.
Review Comment:
Please open a ticket for this with Fixed Version 3.0
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/Types.java:
##########
@@ -158,6 +159,9 @@ public class Types {
/** Returns type information for {@link java.time.Instant}. Supports a null value. */
public static final TypeInformation<Instant> INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO;
+ /** Returns type information for {@link java.util.UUID}. Supports a null value. */
Review Comment:
```suggestion
/** Returns type information for {@link java.util.UUID}. */
```
the serializer code says:
> so there is no reserved value for {@code null}; nullability is handled by wrapping serializers
##########
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/extraction/DataTypeExtractorTest.java:
##########
@@ -81,6 +81,10 @@ private static Stream<TestSpec> testData() {
// simple extraction of BYTES
TestSpec.forType(byte[].class).expectDataType(DataTypes.BYTES()),
+ // automatic extraction of UUID for Table/SQL (DataStream stays opt-in via
+ // Types.UUID)
Review Comment:
unrelated comment here
```suggestion
// automatic extraction of UUID for Table/SQL
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/UuidITCase.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.stream.sql;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.of;
+
+/** Runtime tests for the {@code UUID} type through the full plan/codegen/execution stack. */
+class UuidITCase {
Review Comment:
Never just implement plain ITCases, usually they should extend from a test base. Otherwise a full Flink local cluster is being used instead of performant testing clusters. I would suggest to use SemanticTestBase here or BuiltInFunctionTestBase?
##########
flink-table/flink-table-type-utils/src/main/java/org/apache/flink/table/data/conversion/DataStructureConverters.java:
##########
@@ -197,6 +198,7 @@ public final class DataStructureConverters {
putConverter(LogicalTypeRoot.STRUCTURED_TYPE, RowData.class, identity());
putConverter(LogicalTypeRoot.RAW, byte[].class, RawByteArrayConverter::create);
putConverter(LogicalTypeRoot.RAW, RawValueData.class, identity());
+ putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));
Review Comment:
```suggestion
putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));
putConverter(LogicalTypeRoot.UUID, byte[].class, identity());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3926251151
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/utils/ClassDataTypeConverter.java:
##########
@@ -79,6 +79,7 @@ public final class ClassDataTypeConverter {
addDefaultDataType(
java.time.Period.class, DataTypes.INTERVAL(DataTypes.YEAR(4), DataTypes.MONTH()));
addDefaultDataType(ColumnList.class, DataTypes.DESCRIPTOR());
+ addDefaultDataType(java.util.UUID.class, DataTypes.UUID());
Review Comment:
Not sure what the convention here Is. most of the java classes here have the absolute name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925492717
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/BasicTypeInfo.java:
##########
@@ -338,5 +344,9 @@ private static <X> TypeComparator<X> instantiateComparator(
TYPES.put(BigInteger.class, BIG_INT_TYPE_INFO);
TYPES.put(BigDecimal.class, BIG_DEC_TYPE_INFO);
TYPES.put(Instant.class, INSTANT_TYPE_INFO);
+ // UUID is intentionally not registered here. Automatic reflective extraction of
+ // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+ // avoid silently changing the serializer of existing java.util.UUID fields (currently
+ // handled by Kryo). This is planned to change in the next major Flink version.
Review Comment:
who and how will remember it when next major Flink version starts?
Should we better have a jira with label something like Flink-3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #29087:
URL: https://github.com/apache/flink/pull/29087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925968288
##########
flink-table/flink-table-type-utils/src/main/java/org/apache/flink/table/data/conversion/DataStructureConverters.java:
##########
@@ -197,6 +198,7 @@ public final class DataStructureConverters {
putConverter(LogicalTypeRoot.STRUCTURED_TYPE, RowData.class, identity());
putConverter(LogicalTypeRoot.RAW, byte[].class, RawByteArrayConverter::create);
putConverter(LogicalTypeRoot.RAW, RawValueData.class, identity());
+ putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));
Review Comment:
Added `byte[]` to the inputOutputConvertion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925538177
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/utils/ClassDataTypeConverter.java:
##########
@@ -79,6 +79,7 @@ public final class ClassDataTypeConverter {
addDefaultDataType(
java.time.Period.class, DataTypes.INTERVAL(DataTypes.YEAR(4), DataTypes.MONTH()));
addDefaultDataType(ColumnList.class, DataTypes.DESCRIPTOR());
+ addDefaultDataType(java.util.UUID.class, DataTypes.UUID());
Review Comment:
any specific reason for absolute name here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925960240
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/BasicTypeInfo.java:
##########
@@ -338,5 +344,9 @@ private static <X> TypeComparator<X> instantiateComparator(
TYPES.put(BigInteger.class, BIG_INT_TYPE_INFO);
TYPES.put(BigDecimal.class, BIG_DEC_TYPE_INFO);
TYPES.put(Instant.class, INSTANT_TYPE_INFO);
+ // UUID is intentionally not registered here. Automatic reflective extraction of
+ // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+ // avoid silently changing the serializer of existing java.util.UUID fields (currently
+ // handled by Kryo). This is planned to change in the next major Flink version.
Review Comment:
Created: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-40546
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/Types.java:
##########
@@ -158,6 +159,9 @@ public class Types {
/** Returns type information for {@link java.time.Instant}. Supports a null value. */
public static final TypeInformation<Instant> INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO;
+ /** Returns type information for {@link java.util.UUID}. Supports a null value. */
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925961929
##########
flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/extraction/DataTypeExtractorTest.java:
##########
@@ -81,6 +81,10 @@ private static Stream<TestSpec> testData() {
// simple extraction of BYTES
TestSpec.forType(byte[].class).expectDataType(DataTypes.BYTES()),
+ // automatic extraction of UUID for Table/SQL (DataStream stays opt-in via
+ // Types.UUID)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29087:
URL: https://github.com/apache/flink/pull/29087#issuecomment-5525266619
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbacb85370ad683b4dd12aa77f3b7bba0e51257f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbacb85370ad683b4dd12aa77f3b7bba0e51257f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbacb85370ad683b4dd12aa77f3b7bba0e51257f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3926243716
##########
flink-core/src/main/java/org/apache/flink/api/common/typeinfo/BasicTypeInfo.java:
##########
@@ -338,5 +344,9 @@ private static <X> TypeComparator<X> instantiateComparator(
TYPES.put(BigInteger.class, BIG_INT_TYPE_INFO);
TYPES.put(BigDecimal.class, BIG_DEC_TYPE_INFO);
TYPES.put(Instant.class, INSTANT_TYPE_INFO);
+ // UUID is intentionally not registered here. Automatic reflective extraction of
+ // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+ // avoid silently changing the serializer of existing java.util.UUID fields (currently
+ // handled by Kryo). This is planned to change in the next major Flink version.
Review Comment:
Created: [issues.apache.org/jira/projects/FLINK/issues/FLINK-40546](https://issues.apache.org/jira/projects/FLINK/issues/FLINK-40546)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40490][table] Add runtime serialization and codegen for the UUID type [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29087:
URL: https://github.com/apache/flink/pull/29087#discussion_r3925963572
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/UuidITCase.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.stream.sql;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.of;
+
+/** Runtime tests for the {@code UUID} type through the full plan/codegen/execution stack. */
+class UuidITCase {
Review Comment:
Removed and converted into semantic tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org