You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/08/16 11:51:12 UTC
[PR] [core] Support geospatial data types [paimon]
JingsongLi opened a new pull request, #9251:
URL: https://github.com/apache/paimon/pull/9251
## What changed
- Add `GEOMETRY` and `GEOGRAPHY` data types aligned with Iceberg, including CRS and geography edge algorithms.
- Store geospatial values as WKB and integrate them with Paimon row data, serializers, schema JSON, statistics, Arrow, and Parquet.
- Preserve geospatial logical annotations in Iceberg metadata and native Parquet schemas.
- Add Python type, casting, schema, writer, and documentation support.
- Add native Spark 4.1 SQL support and an explicit compatibility boundary for older Spark versions.
- Expose geospatial values as `VARBINARY` in Flink SQL while preserving the Paimon geospatial schema.
## Why
Iceberg now supports geospatial logical types. Paimon needs compatible schema semantics and storage behavior so geometry and geography columns can be represented and round-tripped across the Java, Python, Iceberg, Parquet, Spark, and Flink integrations.
## User impact
Users can define geometry and geography columns in Paimon schemas, persist WKB values, retain CRS and edge-algorithm metadata, and use the types natively through Spark 4.1. Flink SQL exposes the values as binary because the supported Flink versions do not provide a native geospatial logical type.
## Validation
- Focused Java suites: 351 tests passed.
- Core `TableTestBase` geospatial table test passed.
- Spark 4.1 native Java tests: 2 passed.
- Spark 4.1 SQL geospatial test passed.
- Flink SQL geospatial ITCase passed with both Flink 1.x and Flink 2.x profiles.
- Python geospatial tests passed, including 32 tests and 11 subtests in the related suites.
- Spotless formatting and `git diff --check` passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support geospatial data types [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9251:
URL: https://github.com/apache/paimon/pull/9251#discussion_r3792109171
##########
paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java:
##########
@@ -196,6 +199,16 @@ private static Object toTypeObject(DataType dataType, int fieldId, int depth) {
return timestampLtzPrecision >= 7 ? "timestamptz_ns" : "timestamptz";
case VARIANT:
return "variant";
+ case GEOMETRY:
+ return String.format("geometry(%s)", ((GeometryType) dataType).getCrs());
Review Comment:
Could we validate the geometry CRS against Iceberg's serialized type grammar before enabling Iceberg metadata? `GeometryType` currently accepts any non-empty string, so a valid Paimon schema such as `GEOMETRY('foo)bar')` is serialized here as `geometry(foo)bar)`. Iceberg 1.11's `Types.fromTypeName("geometry(foo)bar)")` rejects that string, which makes the generated metadata unreadable by Iceberg. We already reject commas for geography; please similarly reject `)` for geometry when Iceberg metadata is enabled (and add a regression test). It would also be safer to normalize or reject leading/trailing CRS whitespace because Iceberg trims it while parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support geospatial data types [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9251:
URL: https://github.com/apache/paimon/pull/9251#discussion_r3792155252
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -449,6 +455,106 @@ private static void validateOnlyContainPrimitiveType(
}
}
+ private static void validateGeospatialTypes(
+ TableSchema schema, CoreOptions options, RowType rowType) {
+ boolean hasGeospatial =
+ containsType(
+ rowType,
+ type -> type.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY));
+ if (!hasGeospatial) {
+ return;
+ }
+
+ checkArgument(
+ CoreOptions.FILE_FORMAT_PARQUET.equals(options.formatType()),
+ "Geometry and geography columns require '%s'='parquet', but was '%s'.",
+ CoreOptions.FILE_FORMAT.key(),
+ options.formatType());
+ options.fileFormatPerLevel()
+ .forEach(
+ (level, format) ->
+ checkArgument(
+ CoreOptions.FILE_FORMAT_PARQUET.equals(format),
+ "Geometry and geography columns require parquet at every level, but '%s' contains '%s:%s'.",
+ CoreOptions.FILE_FORMAT_PER_LEVEL.key(),
+ level,
+ format));
+ checkArgument(
+ options.changelogFileFormat() == null
+ || CoreOptions.FILE_FORMAT_PARQUET.equals(options.changelogFileFormat()),
+ "Geometry and geography columns require '%s' to be parquet, but was '%s'.",
+ CoreOptions.CHANGELOG_FILE_FORMAT.key(),
+ options.changelogFileFormat());
+ validateIcebergGeospatialTypes(rowType, options);
+
+ List<String> geospatialClusteringColumns =
+ schema.fields().stream()
+ .filter(field -> options.clusteringColumns().contains(field.name()))
+ .filter(
+ field ->
+ containsType(
+ field.type(),
+ type ->
+ type.isAnyOf(
+ DataTypeRoot.GEOMETRY,
+ DataTypeRoot.GEOGRAPHY)))
+ .map(DataField::name)
+ .collect(Collectors.toList());
+ checkArgument(
+ geospatialClusteringColumns.isEmpty(),
+ "Geometry and geography columns cannot be clustering columns: %s.",
+ geospatialClusteringColumns);
+
+ Set<String> geospatialFields =
+ schema.fields().stream()
+ .filter(
+ field ->
+ field.type()
+ .isAnyOf(
+ DataTypeRoot.GEOMETRY,
+ DataTypeRoot.GEOGRAPHY))
+ .map(DataField::name)
+ .collect(Collectors.toSet());
+ Set<String> geospatialBucketKeys = new HashSet<>(schema.bucketKeys());
+ geospatialBucketKeys.retainAll(geospatialFields);
+ checkArgument(
+ geospatialBucketKeys.isEmpty(),
+ "Geometry and geography columns cannot be bucket keys: %s.",
+ geospatialBucketKeys);
+ Set<String> geospatialSequenceFields = new HashSet<>(options.sequenceField());
Review Comment:
This check only includes fields whose **top-level** type is `GEOMETRY` or `GEOGRAPHY`. A nested geospatial type can therefore still be selected as a sequence field. For example, `seq ROW<geom GEOMETRY>` with `sequence.field=seq` passes `validateTableSchema`, but creating the writer's `UserDefinedSeqComparator` fails in `GenerateUtils.generateRowCompare` with `IllegalArgumentException: Illegal type: GEOMETRY(OGC:CRS84)`.
Could we reject sequence fields whose type recursively contains a geospatial type (similar to the clustering-column check above), and add a regression test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support geospatial data types [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9251:
URL: https://github.com/apache/paimon/pull/9251#discussion_r3792042580
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -449,6 +455,75 @@ private static void validateOnlyContainPrimitiveType(
}
}
+ private static void validateGeospatialTypes(
+ TableSchema schema, CoreOptions options, RowType rowType) {
+ boolean hasGeospatial =
+ containsType(
+ rowType,
+ type -> type.isAnyOf(DataTypeRoot.GEOMETRY, DataTypeRoot.GEOGRAPHY));
+ if (!hasGeospatial) {
+ return;
+ }
+
+ checkArgument(
+ CoreOptions.FILE_FORMAT_PARQUET.equals(options.formatType()),
+ "Geometry and geography columns require '%s'='parquet', but was '%s'.",
+ CoreOptions.FILE_FORMAT.key(),
+ options.formatType());
+ options.fileFormatPerLevel()
+ .forEach(
+ (level, format) ->
+ checkArgument(
+ CoreOptions.FILE_FORMAT_PARQUET.equals(format),
+ "Geometry and geography columns require parquet at every level, but '%s' contains '%s:%s'.",
+ CoreOptions.FILE_FORMAT_PER_LEVEL.key(),
+ level,
+ format));
+ checkArgument(
+ options.changelogFileFormat() == null
+ || CoreOptions.FILE_FORMAT_PARQUET.equals(options.changelogFileFormat()),
+ "Geometry and geography columns require '%s' to be parquet, but was '%s'.",
+ CoreOptions.CHANGELOG_FILE_FORMAT.key(),
+ options.changelogFileFormat());
+ IcebergOptions.StorageType icebergStorage =
+ options.toConfiguration().get(IcebergOptions.METADATA_ICEBERG_STORAGE);
+ if (icebergStorage != IcebergOptions.StorageType.DISABLED) {
+ checkArgument(
+ options.toConfiguration().get(IcebergOptions.FORMAT_VERSION) == 3,
+ "Geometry and geography columns require '%s'='3' when Iceberg metadata is enabled.",
+ IcebergOptions.FORMAT_VERSION.key());
+ checkArgument(
+ icebergStorage != IcebergOptions.StorageType.REST_CATALOG,
+ "Geometry and geography columns do not support '%s'='%s' because the bundled Iceberg REST client cannot parse Iceberg v3 geospatial types.",
+ IcebergOptions.METADATA_ICEBERG_STORAGE.key(),
+ IcebergOptions.StorageType.REST_CATALOG);
+ validateIcebergGeographyCrs(rowType);
+ }
+
+ Set<String> geospatialFields =
+ schema.fields().stream()
+ .filter(
+ field ->
+ field.type()
+ .isAnyOf(
+ DataTypeRoot.GEOMETRY,
+ DataTypeRoot.GEOGRAPHY))
+ .map(DataField::name)
+ .collect(Collectors.toSet());
+ Set<String> geospatialBucketKeys = new HashSet<>(schema.bucketKeys());
+ geospatialBucketKeys.retainAll(geospatialFields);
+ checkArgument(
+ geospatialBucketKeys.isEmpty(),
+ "Geometry and geography columns cannot be bucket keys: %s.",
+ geospatialBucketKeys);
+ Set<String> geospatialSequenceFields = new HashSet<>(options.sequenceField());
+ geospatialSequenceFields.retainAll(geospatialFields);
+ checkArgument(
+ geospatialSequenceFields.isEmpty(),
+ "Geometry and geography columns cannot be sequence fields: %s.",
+ geospatialSequenceFields);
Review Comment:
Could we also reject geospatial columns in `clustering.columns`? At the moment schema validation accepts, for example, an append table with `clustering.incremental=true`, `clustering.columns=geom`, and `clustering.strategy=order`, but the clustering path later builds a `RecordComparator` for this field. `GenerateUtils.generateCompare` has no `GEOMETRY`/`GEOGRAPHY` branch (and `TypeCheckUtils` marks both types as non-comparable), so comparator generation fails with `IllegalArgumentException: Illegal type: GEOMETRY` only when clustering runs. This leaves a valid table configuration that fails at runtime. Please either define an ordering for these types or reject them for both incremental clustering and `pk-clustering-override` during schema validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support geospatial data types [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9251:
URL: https://github.com/apache/paimon/pull/9251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org