You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2026/05/05 19:05:55 UTC
[I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
laskoviymishka opened a new issue, #989:
URL: https://github.com/apache/iceberg-go/issues/989
Parent: #589 (v3 support tracking)
Iceberg v3 adds `geometry` and `geography` as first-class primitive types. WKB ([Well-Known Binary](https://libgeos.org/specifications/wkb/)) is the on-disk encoding; the GeoArrow extension provides the canonical Arrow representation. Spec: [Iceberg primitive types](https://iceberg.apache.org/spec/#primitive-types), [Parquet Geospatial types](https://github.com/apache/parquet-format/blob/master/Geospatial.md).
Two stale WIP PRs exist (#628, #663) but no parent issue tied them together. This umbrella tracks the full surface and breaks it into six independently mergeable pieces. Looking for an owner — happy to mentor the contributor who picks it up.
Sub-issues:
- Type + schema plumbing — small, foundational
- WKB encoding + GeoArrow conversion — small, depends on type plumbing
- Parquet GeoStatistics emission — small, depends on Arrow conversion
- Manifest-level geo bounds aggregation — small, depends on parquet stats
- BBox predicate evaluation + pruning — medium, depends on manifest bounds
- Geo defaults + write path — medium, depends on Arrow conversion + predicates
Each child is sized to ship in isolation. The chain is mostly linear: types → Arrow → parquet stats → manifest bounds → predicates → write. Cross-client coverage matters at every step — Java/pyiceberg fixtures should pin the wire format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-4480211758
Thanks very much for your response! I have made a new issue since I didn't see an existing one for geospatial test fixtures and asked a few follow ups near to not clutter this issue 👍 I will likely start with that and then may sync via slack with others for future PRs in that chain once I do an initial contribution and get a better sense of the overall codebase / current work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6680086033
>> the parquet writer in arrow-go that iceberg-go uses doesn't write geospatial row group statistics
Yep, it’s a gap. arrow-go has the generated thrift struct, but nothing populates it on write or exposes it on read.
On the iceberg-go side we worked around that in #1371 by computing bounds from WKB ourselves for the manifest, so pruning works for files we write. The remaining gap is imports: files added through `AddDataFilesByPath` only go through `DataFileStatsFromMeta`, so they end up without geo bounds.
On @zeroshade’s question: C++ hand-rolls this with a ~250-line `WKBGeometryBounder` in `parquet/geospatial`, no geo dependency, and only for geometry; geography is skipped. Rust has a separate `parquet-geospatial` crate on top of georust `wkb` / `geo-traits`, behind an optional geospatial feature. So both do it in the writer from the logical type, which also points away from adding an extension hook.
My slight preference is to do the same in arrow-go: a small WKB bounder rather than pulling `go-geom` into the parquet module, since Go doesn’t have cargo-style feature gating. For bbox we only need to walk the coordinates, not build a full geometry model.
@zeroshade i'll leave a final dependency call on you.
If this lands, it would also be useful to expose the stats on read, so iceberg-go can pick up bounds from externally written files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "C-Loftus (via GitHub) via issues" <is...@iceberg.apache.org>.
C-Loftus commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6689354216
Thanks for that writeup Andrei. I am in agreement.
> My slight preference is to do the same in arrow-go: a small WKB bounder rather than pulling go-geom into the parquet module, since Go doesn’t have cargo-style feature gating. For bbox we only need to walk the coordinates, not build a full geometry model.
If everyone is in aligned on this, I would be happy to submit a PR for this to arrow-go. I would probably start with just geometry, similar to C++, and then file geography as a follow up issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "C-Loftus (via GitHub) via issues" <is...@iceberg.apache.org>.
C-Loftus commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6642688353
Question @laskoviymishka / @zeroshade , correct me if I am wrong but I am pretty sure at the moment the parquet writer in arrow-go that iceberg-go uses doesn't write geospatial row group statistics.
I would like to add this. However, I was curious if you have any preference on where this belongs?
To my understanding, I can't implement this in geoarrow-go since the only hooks arrow-go exposes into the writer is `ExtensionCustomParquetType.ParquetLogicalType()` (`parquet/pqarrow/schema.go:128`), which chooses the logical type and nothing more. Also the chunk metadata builder's thrift struct is unexported, so an extension type has no way to contribute column-chunk metadata. Adding a new custom hook to compute these statistics and contribute the metadata into the writer would also be an arrow-go change anyways, and it would diverge from the C++ and Rust design where the writer computes statistics from the logical type alone so I'm inclined not to do that.
However, if I contribute this to arrow-go directly then arrow-go is likely to either require pulling in the go-geom dependency for working with WKB or reimplement a fair bit of WKB parsing.
Did either of you have thoughts on the best way to approach this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-4479477750
@C-Loftus, thanks for jumping in!
Honest answer: the sub-issues are mostly a linear chain (#990 → #991 → #992 → #993 → #994 → #995), and the first two are already with @happydave1. So nothing downstream is truly unblocked yet.
That said, there are two good ways to start contributing on geo now:
1. **Cross-client fixtures.** The umbrella issue calls out Java / PyIceberg fixtures to pin the wire format at each step, and I don’t think anyone owns that yet. Building WKB sample tables, Java / PyIceberg round-trip test data, and GeoArrow conformance fixtures would be genuinely useful work. It doesn’t depend on the Go code landing first, and once it exists, every sub-issue can use it. Less glamorous than feature code, but really valuable for a multi-client spec like this.
2. **Pair with @happydave1 on #991.** WKB encoding + GeoArrow conversion is a hard piece of work, and it might make more sense to tackle it together than to split it cleanly on paper. Best place to sync is probably the Apache Iceberg Slack rather than issue comments — feel free to start a thread there and loop @happydave1 in.
Also, reviewing PR #984 as it comes together would be a low-cost way to ramp on the codebase while you decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-4479112617
I would be happy to help implementing geo support. Is there a specific issue that others are not working on and would be straightforward to start with as a new contributor?
Also, should I wait for https://github.com/apache/iceberg-go/pull/984 to be merged before proceeding?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-5375756376
+1 to @C-Loftus -- bumping arrow go seems to be last big thing left open
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-5355669021
@tanmayrauth I believe we still need to bump the arrow-go version due to some of the reasons specified in #1569 Please let me know if I am mistaken though.
- At least from what I'm seeing still, I can build a table with a geometry column, but reading a table with a geometry column written by iceberg-go v0.6.1-0.20260819204557-80630b4df4a4 fails in Scan(...).ToArrowRecords:
`invalid: mismatch storage type 'extension_type<storage=binary>' for extension type 'extension_type<storage=binary>'`
- I believe the cause is that geoarrow-go 20260728 writes the column with the Parquet GEOMETRY(crs=...) logical type. On read, pqarrow rebuilds a `WKBType` from it via `ArrowTypeFromParquet`, which tags the CRS as srid, while the stored ARROW:schema carries authority_code. The types compare unequal, and `applyOriginalStorageMetadata` in arrow-go v18.7.1-0.20260727191739-1b1f82f5848c still expects the inferred type to be bare storage, so it errors.
- From what I can tell, this has been fixed in https://github.com/apache/arrow-go/pull/1051 in the upstream arrow-go library, but there hasn't been a release and thus it hasn't been pulled into iceberg-go yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "zeroshade (via GitHub) via issues" <is...@iceberg.apache.org>.
zeroshade commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6664844144
> it would diverge from the C++ and Rust design where the writer computes statistics from the logical type alone so I'm inclined not to do that.
How do the C++ and Rust implementations handle this? Do they pull in a geospatial dependency or implement the WKB parsing themselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "happydave1 (via GitHub)" <gi...@apache.org>.
happydave1 commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-4382237628
I'm happy to pick up the work for adding Geography and Geometry types! Thanks for breaking the issue up into sub issues!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-5351719622
@laskoviymishka do we have any more items related to this feature or are we good to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: geometry and geography type support (Iceberg v3) [iceberg-go]
Posted by "zeroshade (via GitHub) via issues" <is...@iceberg.apache.org>.
zeroshade commented on issue #989:
URL: https://github.com/apache/iceberg-go/issues/989#issuecomment-6746972927
Sounds good to me, I'd be happy with the small WKB bounder that doesn't need to pull go-geom in. Let's go with it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org