You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2024/05/11 08:43:27 UTC
[PR] WIP: Add geometry type [orc-format]
wgtmac opened a new pull request, #18:
URL: https://github.com/apache/orc-format/pull/18
### What changes were proposed in this pull request?
Add a new geometry type to Apache ORC.
### Why are the changes needed?
Apache Iceberg is adding geospatial support: https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI. It would be good if Apache ORC can support geometry type natively.
### How was this patch tested?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2303881528
I think we can wait until both Java and C++ PoC implementations are finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2238169119
Also, cc @williamhyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2303877093
Yes, I believe so. And I will start PoC implementations later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2787939369
Gentle ping, @wgtmac .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2238245084
@dongjoon-hyun This is not finalized yet. I will update it once ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2788164263
Thank you, @wgtmac .
I'm referring the latest Apache Iceberg community sync-up meeting on March 16th which superseded the above discussion on February 12th.
https://youtu.be/9BBZKTfcU0s?t=44m10s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2303879385
Got it. When do you want to release `orc-format v1.1`?
1. Now to help the PoC?
2. After finishing PoC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #18:
URL: https://github.com/apache/orc-format/pull/18#discussion_r2034293559
##########
src/main/proto/orc/proto/orc_proto.proto:
##########
@@ -84,6 +84,27 @@ message CollectionStatistics {
optional uint64 total_children = 3;
}
+// Bounding box for Geometry or Geography type in the representation of min/max
+// value pair of coordinates from each axis.
+message BoundingBox {
+ optional double xmin = 1;
+ optional double xmax = 2;
+ optional double ymin = 3;
+ optional double ymax = 4;
+ optional double zmin = 5;
+ optional double zmax = 6;
+ optional double mmin = 7;
+ optional double mmax = 8;
+}
+
+// Statistics specific to Geometry or Geography type
+message GeospatialStatistics {
+ // A bounding box of geospatial instances
+ optional BoundingBox bbox = 1;
Review Comment:
Yes, `bbox` is a well-known geospatial acronym.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2791213225
Since this is an improvement without a breaking change, shall we release as Apache ORC v1.1.0? WDYT, @wgtmac ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #18:
URL: https://github.com/apache/orc-format/pull/18#discussion_r2034293027
##########
specification/ORCv2.md:
##########
@@ -301,6 +378,9 @@ message ColumnStatistics {
optional BinaryStatistics binaryStatistics = 8;
optional TimestampStatistics timestampStatistics = 9;
optional bool hasNull = 10;
+ optional uint64 bytes_on_disk = 11;
+ optional CollectionStatistics collection_statistics = 12;
Review Comment:
I've created https://github.com/apache/orc-format/pull/22 to address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2791332043
@dongjoon-hyun Sounds good! I think this can help the implementation a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2781959139
Yes, this is ready for review @dongjoon-hyun
cc @ffacs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2303873882
Thank you for updating. Is this ready, @wgtmac ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2238168172
Thank you, @wgtmac . Is it for Apache ORC 1.1.0? If this is final, I'm +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #18:
URL: https://github.com/apache/orc-format/pull/18#discussion_r2032264096
##########
specification/ORCv2.md:
##########
@@ -301,6 +378,9 @@ message ColumnStatistics {
optional BinaryStatistics binaryStatistics = 8;
optional TimestampStatistics timestampStatistics = 9;
optional bool hasNull = 10;
+ optional uint64 bytes_on_disk = 11;
+ optional CollectionStatistics collection_statistics = 12;
Review Comment:
Can we handle these two above lines independently because this is irrelevant to the `geometry`, @wgtmac ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #18:
URL: https://github.com/apache/orc-format/pull/18#discussion_r2032265481
##########
src/main/proto/orc/proto/orc_proto.proto:
##########
@@ -84,6 +84,27 @@ message CollectionStatistics {
optional uint64 total_children = 3;
}
+// Bounding box for Geometry or Geography type in the representation of min/max
+// value pair of coordinates from each axis.
+message BoundingBox {
+ optional double xmin = 1;
+ optional double xmax = 2;
+ optional double ymin = 3;
+ optional double ymax = 4;
+ optional double zmin = 5;
+ optional double zmax = 6;
+ optional double mmin = 7;
+ optional double mmax = 8;
+}
+
+// Statistics specific to Geometry or Geography type
+message GeospatialStatistics {
+ // A bounding box of geospatial instances
+ optional BoundingBox bbox = 1;
Review Comment:
Is `bbox` a well-known name? I'm just wondering if we can use `bounding_box` like the other field names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2305036669
+1 for the decision. Thank you for the confirmation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2779465722
Thank you. Is this ready, @wgtmac ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2788050356
> Are you sure that Apache Iceberg community keeps Apache ORC support in the next table format?
I'm not sure about the long-term goal. The recent discussion w.r.t the file format API still considers Apache ORC: https://lists.apache.org/thread/ovyh52m2b6c1hrg4fhw3rx92bzr793n2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun merged PR #18:
URL: https://github.com/apache/orc-format/pull/18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry and geography types [orc-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2788223205
Thanks for the info! I just watched that video and understand the concern from the Iceberg community. There are several issues like missing features (variant, geometry, etc.) and implementation (default values, schema evolution, etc.) The the latter issue, does anyone in the Apache ORC community know the detail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1717: Add geometry type [orc-format]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #18:
URL: https://github.com/apache/orc-format/pull/18#issuecomment-2240636648
Thank you! If it's finalized, please send a head-up email to dev@orc once more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org