You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/27 10:15:14 UTC
[I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
alamb opened a new issue, #8717:
URL: https://github.com/apache/arrow-rs/issues/8717
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of https://github.com/apache/arrow-rs/issues/8373
Parquet now has the geometry and geography type - see [`Geospatial.md`](https://github.com/apache/parquet-format/blob/master/Geospatial.md) and the rust reader can read/write them.
However at the moment the arrow reader/writer in the parquet crate returns them as binary columns (I think)
@paleolimbot says in https://github.com/apache/parquet-site/pull/123#discussion_r2455434022:
> In arrow-rs you currently have to try pretty hard to actually get the types to be written or read but it's tested and can be done.
**Describe the solution you'd like**
I would like a better experience of reading / writing geospatial types with arrow-rs and the arrow-reader
**Describe alternatives you've considered**
I think the @kylebarron has some version of this API in the geoparquet crate https://crates.io/crates/geoparquet
However, I am not sure what the
For inspiration, I would suggest an API similar to the support for Variant:
https://docs.rs/parquet/latest/parquet/variant/index.html#example-writing-a-parquet-file-with-variant-column
In that case we defined [`VariantArray`](https://docs.rs/parquet/latest/parquet/variant/struct.VariantArray.html) that wraps a `StructArray` and implements all the Variant details, and then have a way to write that to/from the parquet file with the appropriate metadata. For example:
```rust
// Get a column with Geospatial type:
let GeospatialArray: array = ....;
let field = array.field("data");
let array = ArrayRef::from(array);
// create a RecordBatch with the VariantArray
let schema = Schema::new(vec![field]);
let batch = RecordBatch::try_new(Arc::new(schema), vec![array])?;
// Now you can write the RecordBatch to the Parquet file, as normal
let file = std::fs::File::create("variant.parquet")?;
let mut writer = ArrowWriter::try_new(file, batch.schema(), None)?;
writer.write(&batch)?;
writer.close()?;
```
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8717: [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY`
URL: https://github.com/apache/arrow-rs/issues/8717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3706680893
Can we mark this as complete now that #8943 is merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3729447746
@BlakeOrth Got it! I suppose the ability for "geoarrow.wkb" to be written to/read from Parquet without user intervention was what I envisioned as scope of the `parquet` crate's involvement...the ability to create/consume geoarrow.wkb arrays is something that geoarrow-rs does very well at the moment. Perhaps if there are specifics you envisioned or that I missed we could open a new issue for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3729968992
Since we already use geoarrow-rs I'm personally ambivalent to where the actual array interactions happen. If anyone can think of any maintenance/interoperability benefits that might come from pushing some array operations into this crate I'd be happy to explore those, but there's nothing impactful that I can think of off the top of my head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3711525377
> [@paleolimbot](https://github.com/paleolimbot) I didn't mark this as being closed from [#8943](https://github.com/apache/arrow-rs/pull/8943) because this issue describes being able to interact with (create/read/write) the geospatial types as arrays as opposed to just being able to round-trip the metadata which [#8943](https://github.com/apache/arrow-rs/pull/8943) implemented. If we wanted to delegate that array interaction to external crates, like geoarrow-rs, this probably could be closed with that caveat.
I think this was an oversight on my part -- I did not mean for this ticket to reflect interacting with the types (e.g. the same as we can do with Variant) but just the basic types (which was added in #8943)
Thus, I will close this ticket as complete, and if we want to contemplate adding a more full featured API to manipulate binary types, let's file a follow on ticket.
Thanks @paleolimbot and @BlakeOrth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3470114057
@paleolimbot @kylebarron Have either of you planned time/bandwidth to implement this? If not, I believe I could get some initial implementation going so we can keep this moving forward. I'm not super familiar with this crate, so it might take me longer than other contributors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3711472637
@paleolimbot I didn't mark this as being closed from #8943 because this issue describes being able to interact with (create/read/write) the geospatial types as arrays as opposed to just being able to round-trip the metadata which #8943 implemented. If we wanted to delegate that array interaction to external crates, like geoarrow-rs, this probably could be closed with that caveat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3470254229
I have some upcoming travel and a release of SedonaDB to get out the door in the next three weeks but I'm happy to review before then (or circle back to the implementation after)!
> I'm not super familiar with this crate, so it might take me longer than other contributors.
Hopefully https://github.com/apache/arrow-rs/pull/8408 can help here (it does almost the same thing but with the Variant extension/logical type).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3716622820
`label_issue.py` automatically added labels {'parquet'} from #8943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3451605821
Thank you for opening!
The required conversion between an Arrow extension type and the Parquet logical types are prototyped in https://github.com/apache/arrow-rs/pull/8222 and here are a few relevant comments:
https://github.com/apache/arrow-rs/pull/8222#discussion_r2299643001
https://github.com/apache/arrow-rs/pull/8222#discussion_r2299657546
...and the Arrow C++ implementation:
https://github.com/apache/arrow/blob/f8b20f131a072ef423e81b8a676f42a82255f4ec/cpp/src/parquet/geospatial/util_json_internal.cc#L47-L81
https://github.com/apache/arrow/blob/f8b20f131a072ef423e81b8a676f42a82255f4ec/cpp/src/parquet/geospatial/util_json_internal.cc#L105-L125
The conversion is sufficiently compact (and Parquet-specific) that I think parquet-geospatial is a good spot for it (and I think a circular dependency would incur from using the implementation in `geoarrow-schema`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [geometry] Wire up arrow reader/writer for `GEOMETRY` and `GEOGRAPHY` [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #8717:
URL: https://github.com/apache/arrow-rs/issues/8717#issuecomment-3471210900
I don't have planned time to implement this, and I'm traveling for 3 weeks in November. Also happy to review when I can
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org