You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "happydave1 (via GitHub)" <gi...@apache.org> on 2026/06/19 05:40:57 UTC
[I] feat(table): add projjson CRS support at arrow write level [iceberg-go]
happydave1 opened a new issue, #1230:
URL: https://github.com/apache/iceberg-go/issues/1230
### Feature Request / Improvement
Per the Iceberg spec, CRS must not contain inlined PROJJSON. Instead, the full PROJJSON definition belongs in a table property and the schema CRS should reference it as `projjson:my-property-name`.
iceberg-go already accepts `projjson:...` at the type/schema layer, but the Arrow write path in `icebergCRSToGeoArrowMetadata` explicitly panics for CRS values starting with `projjson:`. To fix this, the write path should resolve `projjson:<property-name>` against table properties and embed the resulting PROJJSON in GeoArrow metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat(table): add projjson CRS support at arrow write level [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on issue #1230:
URL: https://github.com/apache/iceberg-go/issues/1230#issuecomment-4915587211
My preference: don't change the SchemaToArrowSchema/TypeToArrowType signatures, and don't put properties on the visitor either. We already have a channel for exactly this. The read path threads table properties through context (arrow_scanner.go does ctx = internal.WithTableProperties(ctx, as.metadata.Properties()) and parquet_files.go reads it back on the other side). This is the write-side mirror of that same problem, so I'd rather reuse that pattern than invent a second mechanism.
Concretely: resolve the projjson:<name> reference at the boundary via a ctx-aware entry point that pulls props from context and materializes the CRS before we visit. The visitor then runs on already-resolved values, so convertToArrow and VisitGeometry/VisitGeography stay unchanged, icebergCRSToGeoArrowMetadata never learns what a table property is, and the interface plus the other implementers (convertToSubstrait, test visitors) are untouched. The write-path callers all have Properties() on hand, so populating the context is cheap at each site.
Carrying props on the visitor struct (Tanmay's option) works too and keeps it local, but it still needs the props fed in through the exported entry points, and it leaks the properties concept into the visitor for what's a pretty narrow feature. The context route avoids both.
And agreed on the second point: resolve-only at write time, no injecting or defaulting PROJJSON on table creation. That's the caller's job per the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat(table): add projjson CRS support at arrow write level [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #1230: feat(table): add projjson CRS support at arrow write level
URL: https://github.com/apache/iceberg-go/issues/1230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat(table): add projjson CRS support at arrow write level [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #1230:
URL: https://github.com/apache/iceberg-go/issues/1230#issuecomment-4752005387
@laskoviymishka could you please assign me this? I have talked with David and I believe I can work on this in parallel with other Geo support
Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat(table): add projjson CRS support at arrow write level [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #1230:
URL: https://github.com/apache/iceberg-go/issues/1230#issuecomment-4779964616
@happydave1 / @laskoviymishka I've started working on this and have 2 questions on this if you don't mind
> To fix this, the write path should resolve projjson:<property-name> against table properties
Do you have a recommendation for how to solve this other than adding an extra arg to `VisitGeometry`? I tried to pass the table properties but then I get the error below since it doesnt match the `SchemaVisitorPerPrimitiveType` interface anymore. I wasn't clear if I can modify the interface or if it should remain unchanged. To look up the definition in the table properties, I think I need to extend `VisitGeometry` in some capacity.
```go
func (c convertToArrow) VisitGeometry(g iceberg.GeometryType, tableProperties *iceberg.Properties) arrow.Field {
```
```
cannot use convertToArrow{} (value of struct type convertToArrow) as iceberg.SchemaVisitorPerPrimitiveType[arrow.Field] value in variable declaration: convertToArrow does not implement iceberg.SchemaVisitorPerPrimitiveType[arrow.Field] (wrong type for method VisitGeography)
have VisitGeography(iceberg.GeographyType, *iceberg.Properties) arrow.Field
want VisitGeography(iceberg.GeographyType) arrow.Field
```
> Instead, the full PROJJSON definition belongs in a table property and the schema CRS should reference it as projjson:my-property-name.
Is it correct that this is the responsibility of the client application calling iceberg-go to populate the desired PROJJSON definitions into the table metadata themselves? Put another way, my understanding is that I shouldn't add any logic in my PR to default to add any PROJJSON definitions to the table upon creation. Wanted to confirm that.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org