You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/10/01 16:59:04 UTC
[PR] Merge geo spatial [arrow-rs]
etseidl opened a new pull request, #8528:
URL: https://github.com/apache/arrow-rs/pull/8528
# Which issue does this PR close?
**Note: this targets a feature branch, not main**
- Part of #5854.
# Rationale for this change
This brings over changes to handling of geo-spatial statistics introduced by @paleolimbot in #8520.
# What changes are included in this PR?
Primarily adds documentation and tests to changes already made. The only significant change is adding a `Default` implementation for `EdgeInterpolationAlgorithm`.
Still TODO is to decide whether to allow unknown algorithms.
# Are these changes tested?
Yes
# Are there any user-facing changes?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#issuecomment-3362114047
Thanks all. I'll merge this now so it's included in #8530.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395468308
##########
parquet/src/basic.rs:
##########
@@ -928,17 +930,31 @@ enum BoundaryOrder {
// ----------------------------------------------------------------------
// Mirrors thrift enum `EdgeInterpolationAlgorithm`
+// TODO(ets): we need to allow for unknown variants. Either hand code this one, or add a new
+// macro that adds an _Unknown variant.
Review Comment:
If stats are robust to unknown algorithms, then I think we should add an `_Unknown` variant here so older readers can still handle newer files. I'll make that change soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#issuecomment-3362106583
Thanks @martin-g. Yes, the long chain of commits is an artifact of how I've been managing this feature branch. I've been branching off of branches so I can work ahead while awaiting reviews. A git rebase at this point would difficult. I could apply the diffs to a clean check out of the feature branch, but as this is likely the last PR on this branch I don't see the need. It all gets squashed in the end 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#issuecomment-3361009025
@etseidl You may want to rebase to latest `main`. There are 220 commits in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395440011
##########
parquet/src/basic.rs:
##########
@@ -928,17 +930,31 @@ enum BoundaryOrder {
// ----------------------------------------------------------------------
// Mirrors thrift enum `EdgeInterpolationAlgorithm`
+// TODO(ets): we need to allow for unknown variants. Either hand code this one, or add a new
+// macro that adds an _Unknown variant.
Review Comment:
The algorithm is taken into account by the writer when writing the stats...statistics for Geography are in theory safe to use for pruning even if the algorithm is unrecognized (although it's difficult to imagine a situation where this would occur except a corrupted file). I put `UNKNOWN` in the other PR because I couldn't make the `From<>` implementation infallible without it but perhaps you don't have that constraint here.
##########
parquet/src/basic.rs:
##########
@@ -456,9 +457,10 @@ impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for LogicalType {
}
18 => {
let val = GeographyType::read_thrift(&mut *prot)?;
+ let algorithm = val.algorithm.unwrap_or_default();
Review Comment:
I don't mind either way...the interpretation of "the default" is part of the Parquet standard (if unset in Thrift, the interpretation is spherical), my thought was that this would make it so that others don't have to read the spec in order to do the right thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395872299
##########
parquet/src/basic.rs:
##########
@@ -456,9 +457,10 @@ impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for LogicalType {
}
18 => {
let val = GeographyType::read_thrift(&mut *prot)?;
+ let algorithm = val.algorithm.unwrap_or_default();
Review Comment:
> Ah, guess I should have read the spec 😅. I'll leave this as is then. @alamb do you have an opinion here? (relevent section of the spec is [here](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography)).
I agree this seems pretty clear cut:
> If unset, the algorithm defaults to SPHERICAL.
Maybe we could change this to explicitly name SPHERICAL and reference the spec, something like
```suggestion
let algorithm = val.algorithm
// unset algorithm means spherical, per the spec:
// https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography
.unwrap_or(EdgeInterpolationAlgorithm::Spherical)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Merge geo spatial [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#issuecomment-3357290874
cc @paleolimbot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395288538
##########
parquet/src/basic.rs:
##########
@@ -456,9 +457,10 @@ impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for LogicalType {
}
18 => {
let val = GeographyType::read_thrift(&mut *prot)?;
+ let algorithm = val.algorithm.unwrap_or_default();
Review Comment:
This change gives me the most pause. If the value is not set in the thrift, do we really want to set a default or leave it unset and handle that downstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2397948775
##########
parquet/src/schema/printer.rs:
##########
@@ -329,9 +329,20 @@ fn print_logical_and_converted(
LogicalType::Variant {
specification_version,
} => format!("VARIANT({specification_version:?})"),
- LogicalType::Geometry { crs } => format!("GEOMETRY({crs:?})"),
+ LogicalType::Geometry { crs } => {
+ if let Some(crs) = crs {
+ format!("GEOMETRY({crs})")
+ } else {
+ "GEOMETRY".to_string()
+ }
+ }
LogicalType::Geography { crs, algorithm } => {
- format!("GEOGRAPHY({crs:?},{algorithm:?})")
+ let algorithm = algorithm.unwrap_or_default();
+ if let Some(crs) = crs {
+ format!("GEOGRAPHY({algorithm}, {crs})")
Review Comment:
Just pointing out that the order of formatting was switched from crs-algorithm to algorithm-crs. I think this is a beneficial change because the CRS could be a long string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395465852
##########
parquet/src/basic.rs:
##########
@@ -456,9 +457,10 @@ impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for LogicalType {
}
18 => {
let val = GeographyType::read_thrift(&mut *prot)?;
+ let algorithm = val.algorithm.unwrap_or_default();
Review Comment:
Ah, guess I should have read the spec 😅. I'll leave this as is then. @alamb do you have an opinion here? (relevent section of the spec is [here](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Incorporate changes made to geospatial statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8528:
URL: https://github.com/apache/arrow-rs/pull/8528#discussion_r2395292792
##########
parquet/src/basic.rs:
##########
@@ -928,17 +930,31 @@ enum BoundaryOrder {
// ----------------------------------------------------------------------
// Mirrors thrift enum `EdgeInterpolationAlgorithm`
+// TODO(ets): we need to allow for unknown variants. Either hand code this one, or add a new
+// macro that adds an _Unknown variant.
Review Comment:
Here I'm assuming that an unknown algorithm will result in ignoring the stats, so should not be fatal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org