You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2026/06/16 00:37:51 UTC
[PR] feat: support geometry and geography types [iceberg-rust]
wirybeaver opened a new pull request, #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653
## Summary
This draft PR adds Iceberg Geometry/Geography primitive type support by reusing arrow-rs/parquet-geospatial support instead of introducing a local geospatial model.
- Adds GeometryType and GeographyType, using parquet_geospatial::WkbEdges for geography edge interpolation algorithms.
- Converts Geometry/Geography to Arrow WKB extension metadata and enables Parquet geospatial logical type writing.
- Maps Avro, Glue, and HMS representations to bytes/binary.
- Rejects non-null JSON defaults, blocks unsupported partition transforms, and skips byte min/max statistics for spatial values.
## Related issues
Related to #2411 and #1884.
## Test Plan
- cargo test -p iceberg
- cargo check -p iceberg-catalog-glue -p iceberg-catalog-hms
- cargo check -p iceberg-datafusion
- cargo public-api -p iceberg --all-features -ss | diff - crates/iceberg/public-api.txt
- cargo check -p iceberg --locked
- git diff --check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-4715922660
@paleolimbot Thanks for pointing out those edge cases. I will take a closer look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#discussion_r3423334645
##########
crates/iceberg/src/spec/datatypes.rs:
##########
@@ -204,6 +207,157 @@ impl From<MapType> for Type {
}
}
+/// Iceberg geometry type.
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone, Hash, Default)]
+pub struct GeometryType {
+ crs: Option<String>,
+}
+
+impl GeometryType {
+ /// Creates a geometry type with an optional coordinate reference system.
+ pub fn new(crs: Option<String>) -> Result<Self> {
+ Ok(Self {
+ crs: normalize_crs(crs)?,
+ })
+ }
+
+ /// Returns the coordinate reference system, or `None` for the Iceberg default CRS.
+ pub fn crs(&self) -> Option<&str> {
+ self.crs.as_deref()
+ }
+}
+
+/// Iceberg geography type.
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+pub struct GeographyType {
+ crs: Option<String>,
+ algorithm: WkbEdges,
+}
+
+impl Default for GeographyType {
+ fn default() -> Self {
+ Self {
+ crs: None,
+ algorithm: WkbEdges::Spherical,
+ }
+ }
+}
+
+impl Hash for GeographyType {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.crs.hash(state);
+ wkb_edges_as_str(self.algorithm).hash(state);
+ }
+}
+
+impl GeographyType {
+ /// Creates a geography type with an optional coordinate reference system and edge interpolation algorithm.
+ pub fn new(crs: Option<String>, algorithm: WkbEdges) -> Result<Self> {
Review Comment:
The public `iceberg::spec` API exposes `parquet_geospatial::WkbEdges` (`GeographyType::new`/`algorithm()`), coupling the type layer to the parquet-geospatial crate. The set of edge algorithms is defined by the spec itself ([edge-interpolation-algorithm](https://iceberg.apache.org/spec/#edge-interpolation-algorithm)) — could we define an Iceberg-owned enum mirroring the spec and convert to `WkbEdges` only at the Parquet boundary, keeping the type layer format-agnostic?
##########
crates/iceberg/src/spec/values/datum.rs:
##########
@@ -417,8 +417,10 @@ impl Datum {
PrimitiveType::Uuid => {
PrimitiveLiteral::UInt128(u128::from_be_bytes(bytes.try_into()?))
}
- PrimitiveType::Fixed(_) => PrimitiveLiteral::Binary(Vec::from(bytes)),
- PrimitiveType::Binary => PrimitiveLiteral::Binary(Vec::from(bytes)),
+ PrimitiveType::Fixed(_)
+ | PrimitiveType::Binary
+ | PrimitiveType::Geometry(_)
+ | PrimitiveType::Geography(_) => PrimitiveLiteral::Binary(Vec::from(bytes)),
Review Comment:
The single-value path decodes geo bytes as opaque WKB. Per the spec, geo bounds are a single point encoded as `x:y[:z][:m]` little-endian f64s, not WKB ([bound-serialization](https://iceberg.apache.org/spec/#bound-serialization), [bounds-for-geometry-and-geography](https://iceberg.apache.org/spec/#bounds-for-geometry-and-geography)). Both implementations skip spatial bounds today so it's latent, but worth agreeing the bound codec follows the spec point encoding so they don't diverge later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#discussion_r3449493959
##########
crates/iceberg/src/spec/datatypes.rs:
##########
@@ -204,6 +207,157 @@ impl From<MapType> for Type {
}
}
+/// Iceberg geometry type.
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone, Hash, Default)]
+pub struct GeometryType {
+ crs: Option<String>,
+}
+
+impl GeometryType {
+ /// Creates a geometry type with an optional coordinate reference system.
+ pub fn new(crs: Option<String>) -> Result<Self> {
+ Ok(Self {
+ crs: normalize_crs(crs)?,
+ })
+ }
+
+ /// Returns the coordinate reference system, or `None` for the Iceberg default CRS.
+ pub fn crs(&self) -> Option<&str> {
+ self.crs.as_deref()
+ }
+}
+
+/// Iceberg geography type.
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+pub struct GeographyType {
+ crs: Option<String>,
+ algorithm: WkbEdges,
+}
+
+impl Default for GeographyType {
+ fn default() -> Self {
+ Self {
+ crs: None,
+ algorithm: WkbEdges::Spherical,
+ }
+ }
+}
+
+impl Hash for GeographyType {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.crs.hash(state);
+ wkb_edges_as_str(self.algorithm).hash(state);
+ }
+}
+
+impl GeographyType {
+ /// Creates a geography type with an optional coordinate reference system and edge interpolation algorithm.
+ pub fn new(crs: Option<String>, algorithm: WkbEdges) -> Result<Self> {
Review Comment:
Done in 3c8a92e. `GeographyType` now uses an Iceberg-owned `EdgeInterpolationAlgorithm`, and Arrow schema conversion maps to/from `parquet_geospatial::WkbEdges` only at the WKB extension metadata boundary.
##########
crates/iceberg/src/spec/values/datum.rs:
##########
@@ -417,8 +417,10 @@ impl Datum {
PrimitiveType::Uuid => {
PrimitiveLiteral::UInt128(u128::from_be_bytes(bytes.try_into()?))
}
- PrimitiveType::Fixed(_) => PrimitiveLiteral::Binary(Vec::from(bytes)),
- PrimitiveType::Binary => PrimitiveLiteral::Binary(Vec::from(bytes)),
+ PrimitiveType::Fixed(_)
+ | PrimitiveType::Binary
+ | PrimitiveType::Geometry(_)
+ | PrimitiveType::Geography(_) => PrimitiveLiteral::Binary(Vec::from(bytes)),
Review Comment:
Agreed. This PR still omits spatial lower/upper bounds from the Parquet writer, so it does not claim WKB-based bound support. If we add spatial bounds later, the bound codec should use the spec point encoding (`x:y[:z][:m]` as little-endian f64 values), not WKB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5138025249
Reopening because this PR was closed accidentally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5148042835
@paleolimbot I created a new PR #2933 Feel free to take a review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5138034294
@paleolimbot This PR was closed by the stale bot. GitHub does not allow me to reopen it (`viewerCanReopen: false`). Could you please reopen it? The branch is still active and I am continuing the work. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5053031894
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5137757667
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2653: feat: support geometry and geography types
URL: https://github.com/apache/iceberg-rust/pull/2653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support geometry and geography types [iceberg-rust]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #2653:
URL: https://github.com/apache/iceberg-rust/pull/2653#issuecomment-5138083132
I'm not a maintainer here so I can't. You can probably make a new PR (or if that doesn't work, rename the branch and open a new PR). I'm still happy to review but this fell off my radar. If you find a way to reopen with your latest work I will take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org