You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/02/07 04:58:43 UTC
[PR] (Updated) Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
paleolimbot opened a new pull request, #45459:
URL: https://github.com/apache/arrow/pull/45459
### Rationale for this change
The GEOMETRY and GEOGRAPHY logical types are being proposed as an addition to the Parquet format.
### What changes are included in this PR?
This is a continuation of @Kontinuation 's initial PR (#43977), which included:
- Added geometry logical types (printing, serialization, deserialization)
- Added geometry column statistics (serialization, deserialization, writing)
- Support reading/writing parquet files containing geometry columns
Changes after this included:
- Rebasing on the latest apache/arrow
- Split geography/geometry types
- Synchronize the final parameter names (e.g., no more "encoding", "edges" -> "algorithm")
I think we still need a few more for this to be merged (pending the format change vote):
- Update the bounding box logic to implement the "wraparound" bounding boxes where `max > min`
- Handle propagation of the parameters to Arrow (I think we can do this via GeoArrow even without a canonical extension type)
### Are these changes tested?
Yes!
### Are there any user-facing changes?
Yes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2749495657
To confirm, the segfault I mentioned is no longer present with the latest change to remove `GeoCrsContext` handling. What triggered it with an older version of this branch:
```python
# creation of the file
gdf = geopandas.GeoDataFrame(geometry=geopandas.points_from_xy([1,2,3], [1,2,3]), crs="EPSG:3857")
table = pa.table(gdf.to_arrow())
pa.feather.write_feather(table, "test_geometry_3857.arrow")
```
And then writing that Arrow table segfaults if the extension type was enabled:
```python
In [1]: import pyarrow.feather
In [2]: table = pa.feather.read_table("test_geometry_3857.arrow")
In [3]: table.schema
Out[3]:
geometry: binary
-- field metadata --
ARROW:extension:name: 'geoarrow.wkb'
ARROW:extension:metadata: '{"crs": "{\"$schema\":\"https://proj.org/sch' + 2613
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 373
In [4]: import pyarrow.parquet as pq
# works fine when no geoarrow extension type is enabled
In [5]: pq.write_table(table, "test_geometry_from_table_no_ext.parquet")
In [6]: import geoarrow.pyarrow as ga
In [7]: table = pa.feather.read_table("test_geometry_3857.arrow")
In [8]: table.schema
Out[8]:
geometry: extension<geoarrow.wkb<WkbType>>
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 373
# segfaults when it is enabled
In [9]: pq.write_table(table, "test_geometry_from_table_ext.parquet")
terminate called after throwing an instance of 'parquet::ParquetException'
what(): Crs encoding 'unknown' is not suppored by GeoCrsContext
Aborted (core dumped)
```
Not sure if this is related to geopandas creating invalid crs metadata? (the bug you fixed on geopandas main)
But so in any case, the above now works without segfault on the latest version of this branch.
This difference makes me wonder one other thing, though, which is still present: for the Parquet code to "see" a geometry column in the Arrow data it is writing, it needs to be an actual _registered_ extension type, and having the extension metadata in the field metadata is not sufficient? (purely for the spec, the metadata is what defines the extension type, and so someone roundtripping Parquet data (e.g. reading in, doing some filtering, writing out again) would loose this type information if they do not have the extension type registered)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016119879
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read coordinate sequence of ", total_bytes,
+ " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
Review Comment:
```suggestion
if (ARROW_PREDICT_FALSE(size_ < sizeof(T))) {
```
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read coordinate sequence of ", total_bytes,
+ " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
Review Comment:
Is it useful or can the `bit_util` function be called directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016137824
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ const std::string& serialized_data);
Review Comment:
Should this take a `std::string_view` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057832331
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
Review Comment:
```suggestion
bool geospatial_types_present() const { return !geospatial_types.empty(); }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057931531
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
We don't need JSON if `PARQUET_MINIMAL_DEPENDENCY` is `ON`, right?
Then, we don't need the following:
```diff
diff --git a/cpp/cmake_modules/DefineOptions.cmake b/cpp/cmake_modules/DefineOptions.cmake
index 16360929a9..93a7ba6063 100644
--- a/cpp/cmake_modules/DefineOptions.cmake
+++ b/cpp/cmake_modules/DefineOptions.cmake
@@ -370,7 +370,8 @@ takes precedence over ccache if a storage backend is configured" ON)
"Build the Parquet libraries"
OFF
DEPENDS
- ARROW_IPC)
+ ARROW_IPC
+ ARROW_JSON)
define_option(ARROW_ORC
"Build the Arrow ORC adapter"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016723333
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
Is there a specific reason? It complicates maintenance to have optional functionality like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007976080
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
Review Comment:
I'll leave the full discussion around this for the `NaN` comment you had below, but I think it's best to check all the things here (i.e., if this object contains any shred of useful information, include it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007970230
##########
cpp/src/parquet/properties.h:
##########
@@ -69,6 +69,45 @@ constexpr int32_t kDefaultThriftContainerSizeLimit = 1000 * 1000;
// PARQUET-978: Minimize footer reads by reading 64 KB from the end of the file
constexpr int64_t kDefaultFooterReadSize = 64 * 1024;
+/// \brief Convert Coordinate Reference System values for GEOMETRY and GEOGRAPHY logical
+/// types
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoCrsContext {
+ public:
+ virtual ~GeoCrsContext() = default;
+
+ /// \brief Remove any previously saved PROJJSON CRS fields
+ virtual void Clear() = 0;
+
+ /// \brief Given a coordinate reference system value and encoding from GeoArrow
+ /// extension metadata, return the value that should be placed in the
+ /// LogicalType::Geography|Geometry(crs=) field
+ ///
+ /// For PROJJSON Crses (the most common way coordinate reference systems arrive
+ /// in Arrow), the Parquet specification forces us to write them to the file
+ /// metadata. The default GeoCrsContext will record such values and return the required
+ /// string that can be placed into the 'crs' of the Geometry or Geography logical
+ /// type.
+ virtual std::string GetParquetCrs(std::string crs_value,
+ const std::string& crs_encoding) = 0;
+
+ /// \brief Returns true if any converted CRS values were PROJJSON whose values are
+ /// stored in this object and should be written to the file metadata
+ virtual bool HasProjjsonCrsFields() = 0;
+
+ /// \brief Add any stored PROJJSON values to the supplied file KeyValueMetadata
+ ///
+ /// The default implementation will add any PROJJSON values it encountered to the file
+ /// KeyValueMetadata; however, a custom implementation may choose to store these values
+ /// somewhere else and implement this method as a no-op.
+ virtual void AddProjjsonCrsFieldsToFileMetadata(
Review Comment:
I removed the GeoCrsContext!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2058332513
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1826,6 +1870,12 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
page_statistics_->IncrementNullCount(num_chunk_levels - non_null_count);
page_statistics_->IncrementNumValues(non_null_count);
page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false);
+
+ if (chunk_geospatial_statistics_ != nullptr) {
+ throw ParquetException(
+ "Writing dictionary-encoded GEOMETRY or GEOGRAPHY with statistics is not "
+ "supported");
+ }
Review Comment:
Is this something we plan to later support?
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
Review Comment:
Is the docstring correct? How does a method named "dimension_empty" return "true if the dimension is valid and any non-NaN values were encountered"?
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Returns false if a bound was explicitly not calculated by a writer when
+ /// importing statistics from thrift
Review Comment:
I don't think I understand what this means. Why would a writer import statistics from thrift?
##########
cpp/src/parquet/geospatial/util_internal.h:
##########
@@ -0,0 +1,216 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <array>
+#include <cmath>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging_internal.h"
+#include "parquet/platform.h"
+
+namespace parquet::geospatial {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct PARQUET_EXPORT BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const;
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Throws ParquetException for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ void MergeGeometry(std::string_view bytes_wkb);
Review Comment:
Can also expose an overload taking a `util:span<const uint8_t>` for convenience.
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
Review Comment:
Can pass a string view for flexibility and to avoid internal copies when calling `substr`
```suggestion
::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
std::string_view crs,
const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
```
##########
python/pyarrow/_parquet.pxd:
##########
@@ -205,6 +224,18 @@ cdef extern from "parquet/api/schema.h" namespace "parquet" nogil:
ParquetCipher algorithm
AadMetadata aad
+
+# Specific array<> types needed for GeoStatistics
+cdef extern from "<array>" namespace "std" nogil:
Review Comment:
You don't have to do it obviously, but for the record it would be relatively easy to upstream `std::array` declarations in Cython, here is an example related PR: https://github.com/cython/cython/pull/6652
##########
cpp/src/parquet/api/reader.h:
##########
@@ -22,6 +22,7 @@
#include "parquet/column_scanner.h"
#include "parquet/exception.h"
#include "parquet/file_reader.h"
+#include "parquet/geospatial/statistics.h"
Review Comment:
I'm not sure we want to further expand the `api.h` header. Every additional inclusion here can increase compile times for users, we should encourage them to include the individual headers as-needed instead.
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,174 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(std::shared_ptr<const LogicalType> type, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(
+ PrimitiveNode::Make("g", Repetition::REQUIRED, type, Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64);
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer =
+ ::arrow::internal::checked_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointXYSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+ std::memcpy(ptr, item.data(), item.size());
+ values[k].len = test::kWkbPointXYSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointXYSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ for (int k = 0; k < kNumRows; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+
+ ASSERT_OK(builder.Append(item));
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(),
+ /*leaf_field_nullable=*/true));
+ }
+
+ void TestWriteAndRead(std::shared_ptr<const LogicalType> type, bool write_arrow) {
+ ASSERT_NO_FATAL_FAILURE(WriteTestData(type, write_arrow));
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ ASSERT_TRUE(type->Equals(*metadata->schema()->Column(0)->logical_type()));
+
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ int64_t start_index = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geo_stats = column_chunk_metadata->geo_statistics();
+ ASSERT_NO_FATAL_FAILURE(CheckGeoStatistics(type, geo_stats, start_index,
+ row_group_metadata->num_rows()));
+ start_index += row_group_metadata->num_rows();
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ auto xy = test::GetWKBPointCoordinateXY(value);
+ EXPECT_TRUE(xy.has_value());
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_EQ(*xy, (std::pair<double, double>(expected_x, expected_y)));
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeoStatistics(std::shared_ptr<const LogicalType> type,
+ std::shared_ptr<geospatial::GeoStatistics> geom_stats,
+ int64_t start_index, int64_t num_rows) {
+ // We don't yet generate statistics for Geography
+ if (type->is_geography()) {
+ ASSERT_EQ(geom_stats, nullptr);
+ return;
+ }
+
+ ASSERT_NE(geom_stats, nullptr);
+ // We wrote exactly one geometry type (POINT, which has code 1)
+ EXPECT_THAT(*geom_stats->geometry_types(), ::testing::ElementsAre(1));
+
+ double expected_xmin = static_cast<double>(start_index);
+ double expected_xmax = expected_xmin + num_rows - 1;
+ double expected_ymin = expected_xmin + 1;
+ double expected_ymax = expected_xmax + 1;
+
+ EXPECT_EQ(geom_stats->lower_bound()[0], expected_xmin);
+ EXPECT_EQ(geom_stats->upper_bound()[0], expected_xmax);
+ EXPECT_EQ(geom_stats->lower_bound()[1], expected_ymin);
+ EXPECT_EQ(geom_stats->upper_bound()[1], expected_ymax);
+ EXPECT_THAT(geom_stats->dimension_valid(),
+ ::testing::ElementsAre(true, true, false, false));
+ }
+
+ protected:
+ std::shared_ptr<Buffer> file_buf;
+};
+
+TEST_F(TestGeometryLogicalType, TestWriteGeometry) {
+ TestWriteAndRead(GeometryLogicalType::Make("srid:1234"), /*write_arrow=*/false);
+}
+
+TEST_F(TestGeometryLogicalType, TestWriteArrowAndReadGeometry) {
+ TestWriteAndRead(GeometryLogicalType::Make("srid:1234"), /*write_arrow=*/true);
+}
+
+TEST_F(TestGeometryLogicalType, TestWriteGeography) {
+ TestWriteAndRead(GeographyLogicalType::Make("srid:1234"), /*write_arrow=*/false);
Review Comment:
Why not also test with `write_arrow=true`?
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax))) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ geospatial::BoundingBox box;
+
+ if (encoded.xy_bounds_present) {
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+ } else {
+ box.min[0] = kNaN;
+ box.max[0] = kNaN;
+ box.min[1] = kNaN;
+ box.max[1] = kNaN;
+ }
+
+ if (encoded.z_bounds_present) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ } else {
+ box.min[2] = kNaN;
+ box.max[2] = kNaN;
+ }
+
+ if (encoded.m_bounds_present) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ } else {
+ box.min[3] = kNaN;
+ box.max[3] = kNaN;
+ }
+
+ bounder_.MergeBox(box);
+ bounder_.MergeGeometryTypes(encoded.geospatial_types);
+ has_geometry_types_ = has_geometry_types_ && encoded.geospatial_types_present();
+ }
+
+ bool is_wraparound_x() const {
+ return is_wraparound(lower_bound()[0], upper_bound()[0]);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ bool bounds_empty() const {
+ for (int i = 0; i < kMaxDimensions; i++) {
+ if (!bound_empty(i)) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ bool bound_empty(int i) const {
+ return std::isinf(bounder_.Bounds().min[i] - bounder_.Bounds().max[i]);
+ }
+
+ bool bound_valid(int i) const {
+ return !std::isnan(bounder_.Bounds().min[i]) && !std::isnan(bounder_.Bounds().max[i]);
+ }
+
+ const std::array<double, kMaxDimensions>& lower_bound() const {
+ return bounder_.Bounds().min;
+ }
+
+ const std::array<double, kMaxDimensions>& upper_bound() const {
+ return bounder_.Bounds().max;
+ }
+
+ std::optional<std::vector<int32_t>> geometry_types() const {
+ if (has_geometry_types_) {
+ return bounder_.GeometryTypes();
+ } else {
+ return std::nullopt;
+ }
+ }
+
+ private:
+ geospatial::WKBGeometryBounder bounder_;
+ bool is_valid_ = true;
+ bool has_geometry_types_ = true;
+
+ template <typename ArrayType>
+ void UpdateArrayImpl(const ::arrow::Array& values) {
+ const auto& binary_array = static_cast<const ArrayType&>(values);
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
+ if (!binary_array.IsNull(i)) {
+ try {
+ bounder_.MergeGeometry(binary_array.GetView(i));
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+ }
+
+ static bool is_wraparound(double dmin, double dmax) {
+ return !std::isinf(dmin - dmax) && dmin > dmax;
+ }
+};
+
+GeoStatistics::GeoStatistics() : impl_(std::make_unique<GeoStatisticsImpl>()) {}
+
+GeoStatistics::GeoStatistics(const EncodedGeoStatistics& encoded) : GeoStatistics() {
+ Decode(encoded);
+}
+
+GeoStatistics::~GeoStatistics() = default;
+
+bool GeoStatistics::Equals(const GeoStatistics& other) const {
+ return impl_->Equals(*other.impl_);
+}
+
+void GeoStatistics::Merge(const GeoStatistics& other) { impl_->Merge(*other.impl_); }
+
+void GeoStatistics::Update(const ByteArray* values, int64_t num_values) {
+ impl_->Update(values, num_values);
+}
+
+void GeoStatistics::UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ impl_->UpdateSpaced(values, valid_bits, valid_bits_offset, num_spaced_values,
+ num_values);
+}
+
+void GeoStatistics::Update(const ::arrow::Array& values) { impl_->Update(values); }
+
+void GeoStatistics::Reset() { impl_->Reset(); }
+
+bool GeoStatistics::is_valid() const { return impl_->is_valid(); }
+
+std::optional<EncodedGeoStatistics> GeoStatistics::Encode() const {
+ if (is_valid()) {
+ return impl_->Encode();
+ } else {
+ return std::nullopt;
+ }
+}
+
+void GeoStatistics::Decode(const EncodedGeoStatistics& encoded) {
+ impl_->Reset();
+ impl_->Update(encoded);
+}
+
+std::array<double, 4> GeoStatistics::lower_bound() const { return impl_->lower_bound(); }
+
+std::array<double, 4> GeoStatistics::upper_bound() const { return impl_->upper_bound(); }
+
+std::array<bool, 4> GeoStatistics::dimension_empty() const {
+ return {impl_->bound_empty(0), impl_->bound_empty(1), impl_->bound_empty(2),
+ impl_->bound_empty(3)};
+}
+
+std::array<bool, 4> GeoStatistics::dimension_valid() const {
+ return {impl_->bound_valid(0), impl_->bound_valid(1), impl_->bound_valid(2),
+ impl_->bound_valid(3)};
+}
+
+std::optional<std::vector<int32_t>> GeoStatistics::geometry_types() const {
+ return impl_->geometry_types();
+}
+
+std::string GeoStatistics::ToString() const {
+ if (!is_valid()) {
+ return "GeoStatistics <invalid>\n";
+ }
+
+ std::stringstream ss;
+ ss << "GeoStatistics " << std::endl;
Review Comment:
Multi-line string representations are not very easy to insert into logs, etc.
Perhaps instead have something like `GeoStatistics<x: ..., y: ..., m: ...>`?
##########
cpp/src/parquet/geospatial/util_internal_test.cc:
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "parquet/geospatial/util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geospatial {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox\n x: [inf, -inf]\n y: [inf, -inf]\n z: [inf, -inf]\n m: "
+ "[inf, -inf]\n");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+ EXPECT_EQ(box_xyzm.ToString(),
+ "BoundingBox\n x: [-10, 10]\n y: [-20, 20]\n z: [-3, 3]\n m: "
+ "[-4, 4]\n");
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ double nan_dbl = std::numeric_limits<double>::quiet_NaN();
+ BoundingBox box_nan({nan_dbl, nan_dbl, nan_dbl, nan_dbl},
+ {nan_dbl, nan_dbl, nan_dbl, nan_dbl});
+ box_xyzm.Merge(box_nan);
+ for (int i = 0; i < 4; i++) {
+ EXPECT_TRUE(std::isnan(box_xyzm.min[i]));
+ EXPECT_TRUE(std::isnan(box_xyzm.max[i]));
+ }
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType geometry_type, Dimensions dimensions,
+ const std::vector<uint8_t>& wkb, const std::vector<double>& box_values = {})
+ : geometry_type(geometry_type), dimensions(dimensions), wkb(wkb) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::kXYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
Review Comment:
Why not put this in `util_internal.h`?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,136 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types (experimental)"""
+
+ def __init__(self):
+ raise TypeError(f"Do not call {self.__class__.__name__}'s constructor directly")
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
Review Comment:
Can you add docstrings for all these properties?
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
Review Comment:
Can make these `inline` IMHO and remove the `static`.
```suggestion
/// \brief The maximum number of dimensions represented by a geospatial type
/// (i.e., X, Y, Z, and M)
inline constexpr int kMaxDimensions = 4;
/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
/// be applied (e.g., because a writer did not provide bounds for a given dimension)
inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
```
See https://stackoverflow.com/a/54467111
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ const std::string kSridPrefix{"srid:"};
+ const std::string kProjjsonPrefix{"projjson:"};
+
+ // Two reccomendataions are explicitly mentioned in the Parquet format for the
Review Comment:
```suggestion
// Two recommendations are explicitly mentioned in the Parquet format for the
```
##########
cpp/src/parquet/column_writer_test.cc:
##########
@@ -1849,8 +1867,158 @@ TEST_F(TestValuesWriterInt32Type, AvoidCompressedInDataPageV2) {
verify_only_one_uncompressed_page(/*total_num_values=*/1);
}
}
-
#endif
+// Test writing and reading geometry columns
+class TestGeometryValuesWriter : public TestPrimitiveWriter<ByteArrayType> {
+ public:
+ void SetUpSchema(Repetition::type repetition, int num_columns) override {
+ std::vector<schema::NodePtr> fields;
+
+ for (int i = 0; i < num_columns; ++i) {
+ std::string name = TestColumnName(i);
+ std::shared_ptr<const LogicalType> logical_type =
+ GeometryLogicalType::Make("srid:1234");
+ fields.push_back(schema::PrimitiveNode::Make(name, repetition, logical_type,
+ ByteArrayType::type_num));
+ }
+ node_ = schema::GroupNode::Make("schema", Repetition::REQUIRED, fields);
+ schema_.Init(node_);
+ }
+
+ void GenerateData(int64_t num_values, uint32_t seed = 0) {
+ values_.resize(num_values);
+
+ buffer_.resize(num_values * kWkbPointXYSize);
+ uint8_t* ptr = buffer_.data();
+ for (int k = 0; k < num_values; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+ std::memcpy(ptr, item.data(), item.size());
+ values_[k].len = kWkbPointXYSize;
+ values_[k].ptr = ptr;
+ ptr += kWkbPointXYSize;
+ }
+
+ values_ptr_ = values_.data();
+ }
+};
+
+TEST_F(TestGeometryValuesWriter, TestWriteAndRead) {
+ this->SetUpSchema(Repetition::REQUIRED, 1);
+ this->GenerateData(SMALL_SIZE);
+ size_t num_values = this->values_.size();
+ auto writer = this->BuildWriter(num_values, ColumnProperties());
+ writer->WriteBatch(this->values_.size(), nullptr, nullptr, this->values_.data());
+
+ writer->Close();
+ this->ReadColumn();
+ for (size_t i = 0; i < num_values; i++) {
+ const ByteArray& value = this->values_out_[i];
+ auto xy = GetWKBPointCoordinateXY(value);
+ EXPECT_TRUE(xy.has_value());
+ auto expected_x = static_cast<double>(i);
+ auto expected_y = static_cast<double>(i + 1);
+ EXPECT_EQ(*xy, (std::pair<double, double>(expected_x, expected_y)));
Review Comment:
Nit: I think you can simply write
```suggestion
EXPECT_EQ(*xy, std::pair{expected_x, expected_y});
```
or
```suggestion
EXPECT_EQ(*xy, std::make_pair(expected_x, expected_y));
```
##########
cpp/src/parquet/types.cc:
##########
@@ -479,6 +480,27 @@ std::shared_ptr<const LogicalType> LogicalType::FromThrift(
return UUIDLogicalType::Make();
} else if (type.__isset.FLOAT16) {
return Float16LogicalType::Make();
+ } else if (type.__isset.GEOMETRY) {
+ std::string crs;
+ if (type.GEOMETRY.__isset.crs) {
+ crs = type.GEOMETRY.crs;
+ }
+
+ return GeometryLogicalType::Make(crs);
+ } else if (type.__isset.GEOGRAPHY) {
+ std::string crs;
+ if (type.GEOGRAPHY.__isset.crs) {
+ crs = type.GEOGRAPHY.crs;
+ }
+
+ LogicalType::EdgeInterpolationAlgorithm algorithm;
+ if (!type.GEOGRAPHY.__isset.algorithm) {
+ algorithm = LogicalType::EdgeInterpolationAlgorithm::SPHERICAL;
+ } else {
+ algorithm = ::parquet::FromThrift(type.GEOGRAPHY.algorithm);
+ }
+
+ return GeographyLogicalType::Make(crs, algorithm);
Review Comment:
```suggestion
return GeographyLogicalType::Make(std::move(crs), algorithm);
```
##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -0,0 +1,238 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_internal.h"
+
+#include <sstream>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geospatial {
+
+std::string BoundingBox::ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+}
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ uint8_t ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ uint32_t ReadUInt32(bool swap) {
+ auto value = ReadChecked<uint32_t>();
+ if (swap) {
+ return ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ void ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
Review Comment:
Why is it called `ReadDoubles`? `ReadCoords` perhaps?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -450,6 +590,24 @@ cdef class ColumnChunkMetaData(_Weakrefable):
statistics.init(self.metadata.statistics(), self)
return statistics
+ @property
+ def is_geo_stats_set(self):
+ """Whether or not geometry statistics are present in metadata (bool)."""
+ return self.metadata.is_geo_stats_set()
+
+ @property
+ def geo_statistics(self):
+ """Statistics for column chunk (:class:`GeoStatistics`)."""
+ if not self.metadata.is_geo_stats_set():
+ return None
+
+ if not self.metadata.geo_statistics().get().is_valid():
+ return None
+
+ cdef GeoStatistics geo_statistics = GeoStatistics.__new__(GeoStatistics)
+ geo_statistics.init(self.metadata.geo_statistics(), self)
+ return geo_statistics
Review Comment:
This is a bit verbose and doing redundant work. How about something like:
```suggestion
c_geo_statistics = self.metadata.geo_statistics()
if not c_geo_statistics or not c_geo_statistics.get().is_valid():
return None
cdef GeoStatistics geo_statistics = GeoStatistics.__new__(GeoStatistics)
geo_statistics.init(c_geo_statistics, self)
return geo_statistics
```
##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -0,0 +1,238 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_internal.h"
+
+#include <sstream>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geospatial {
+
+std::string BoundingBox::ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+}
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ uint8_t ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ uint32_t ReadUInt32(bool swap) {
+ auto value = ReadChecked<uint32_t>();
+ if (swap) {
+ return ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ void ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ throw ParquetException("Can't read coordinate sequence of ", total_bytes,
+ " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (swap) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (auto& c : coord) {
+ c = ::arrow::bit_util::ByteSwap(c);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
Review Comment:
Why not use a `util::span<const uint8_t>`?
##########
cpp/src/parquet/geospatial/util_internal_test.cc:
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "parquet/geospatial/util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geospatial {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox\n x: [inf, -inf]\n y: [inf, -inf]\n z: [inf, -inf]\n m: "
+ "[inf, -inf]\n");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+ EXPECT_EQ(box_xyzm.ToString(),
+ "BoundingBox\n x: [-10, 10]\n y: [-20, 20]\n z: [-3, 3]\n m: "
+ "[-4, 4]\n");
Review Comment:
If you don't check that some bounding boxes are unequal, then you may miss bugs in `operator==` or `operator!=`.
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Returns false if a bound was explicitly not calculated by a writer when
+ /// importing statistics from thrift
+ ///
+ /// When calculating statistics, this will always be true because this implementation
+ /// calculates statistics for all dimensions.
+ std::array<bool, kMaxDimensions> dimension_valid() const;
Review Comment:
Can you also explain the relationship between `dimension_empty` and `dimension_valid`? It's not clear to me.
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
Review Comment:
Nit: make this more readable by making the initialization value more explicit?
```suggestion
bool xy_bounds_present{false};
```
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Returns false if a bound was explicitly not calculated by a writer when
+ /// importing statistics from thrift
+ ///
+ /// When calculating statistics, this will always be true because this implementation
+ /// calculates statistics for all dimensions.
+ std::array<bool, kMaxDimensions> dimension_valid() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates. Returns
+ /// std::nullopt if the writer did not calculate geometry types for statistics
+ /// imported from Thrift.
Review Comment:
Same question: what does the writer have to do with importing statistics from Thrift?
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ const std::string kSridPrefix{"srid:"};
+ const std::string kProjjsonPrefix{"projjson:"};
+
+ // Two reccomendataions are explicitly mentioned in the Parquet format for the
+ // LogicalType crs:
+ //
+ // - "srid:XXXX" as a way to encode an application-specific integer identifier
+ // - "projjson:some_field_name" as a way to avoid repeating PROJJSON strings
+ // unnecessarily (with a suggestion to place them in the file metadata)
+ //
+ // While we don't currently generate those values to reduce the complexity
+ // of the writer, we do interpret these values according to the suggestion in
+ // the format and pass on this information to GeoArrow.
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (::arrow::internal::StartsWith(crs, kSridPrefix)) {
+ return R"("crs": ")" + crs.substr(kSridPrefix.size()) + R"(", "crs_type": "srid")";
+ } else if (::arrow::internal::StartsWith(crs, kProjjsonPrefix)) {
+ std::string metadata_field = crs.substr(kProjjsonPrefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
Review Comment:
Do we want to ensure that `projjson_value` is actually valid JSON?
##########
cpp/src/parquet/geospatial/util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string_view>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// GeoArrow `ARROW:extension:metadata` (JSON-encoded extension type metadata)
+///
+/// Returns the appropriate LogicalType, Invalid if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
Review Comment:
Let's remove the ARROW_JSON mention?
##########
cpp/src/parquet/geospatial/util_internal_test.cc:
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "parquet/geospatial/util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geospatial {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox\n x: [inf, -inf]\n y: [inf, -inf]\n z: [inf, -inf]\n m: "
+ "[inf, -inf]\n");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+ EXPECT_EQ(box_xyzm.ToString(),
+ "BoundingBox\n x: [-10, 10]\n y: [-20, 20]\n z: [-3, 3]\n m: "
+ "[-4, 4]\n");
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ double nan_dbl = std::numeric_limits<double>::quiet_NaN();
+ BoundingBox box_nan({nan_dbl, nan_dbl, nan_dbl, nan_dbl},
+ {nan_dbl, nan_dbl, nan_dbl, nan_dbl});
+ box_xyzm.Merge(box_nan);
+ for (int i = 0; i < 4; i++) {
+ EXPECT_TRUE(std::isnan(box_xyzm.min[i]));
+ EXPECT_TRUE(std::isnan(box_xyzm.max[i]));
+ }
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType geometry_type, Dimensions dimensions,
+ const std::vector<uint8_t>& wkb, const std::vector<double>& box_values = {})
+ : geometry_type(geometry_type), dimensions(dimensions), wkb(wkb) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::kXYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderBounds) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_NO_THROW(bounder.MergeGeometry(
+ {reinterpret_cast<const char*>(item.wkb.data()), item.wkb.size()}));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
+ uint32_t wkb_type =
+ static_cast<int>(item.dimensions) * 1000 + static_cast<int>(item.geometry_type);
+ EXPECT_THAT(bounder.GeometryTypes(), ::testing::ElementsAre(::testing::Eq(wkb_type)));
+
+ bounder.Reset();
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+ EXPECT_TRUE(bounder.GeometryTypes().empty());
+}
+
+TEST_P(WKBTestFixture, TestWKBBounderErrorForTruncatedInput) {
+ auto item = GetParam();
+ WKBGeometryBounder bounder;
+
+ // Make sure an error occurs for any version of truncated input to the bounder
Review Comment:
Apparently we should also check that a too large input should throw as well?
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1090,6 +1094,49 @@ TEST_F(TestConvertParquetSchema, ParquetSchemaArrowUuidExtension) {
}
}
+TEST_F(TestConvertParquetSchema, ParquetSchemaGeoArrowExtensions) {
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ {
+ // Parquet file does not contain Arrow schema.
+ // By default, both fields should be treated as binary() fields in Arrow.
+ auto arrow_schema = ::arrow::schema({::arrow::field("geometry", BINARY, true),
+ ::arrow::field("geography", BINARY, true)});
+ std::shared_ptr<KeyValueMetadata> metadata{};
+ ASSERT_OK(ConvertSchema(parquet_fields, metadata));
+ CheckFlatSchema(arrow_schema);
+ }
+
+ {
+ // Parquet file does not contain Arrow schema.
+ // If Arrow extensions are enabled and extensions are registered,
+ // fields will be interpreted as geoarrow_wkb(binary()) extension fields.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ ArrowReaderProperties props;
+ props.set_arrow_extensions_enabled(true);
+ auto arrow_schema = ::arrow::schema(
+ {::arrow::field(
+ "geometry",
+ test::geoarrow_wkb(R"({"crs": "OGC:CRS84", "crs_type": "authority_code"})"),
+ true),
+ ::arrow::field(
+ "geography",
+ test::geoarrow_wkb(
+ R"({"crs": "OGC:CRS84", "crs_type": "authority_code", "edges": "spherical"})"),
+ true)});
+ std::shared_ptr<KeyValueMetadata> metadata{};
+ ASSERT_OK(ConvertSchema(parquet_fields, metadata, props));
+ CheckFlatSchema(arrow_schema);
+ }
Review Comment:
Can you also add a test for when Arrow extensions are enabled but the extension type isn't registered?
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1343,6 +1390,96 @@ TEST_F(TestConvertArrowSchema, ParquetFlatPrimitivesAsDictionaries) {
ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
}
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsLonLat) {
+ // All the Arrow Schemas below should convert to the type defaults for GEOMETRY
+ // and GEOGRAPHY when GeoArrow extension types are registered and the appropriate
+ // writer option is set.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ // There are several ways that longitude/latitude could be specified when coming from
+ // GeoArrow, which allows null, missing, arbitrary strings (e.g., Authority:Code), and
+ // PROJJSON.
+ std::vector<std::string> geoarrow_lonlat = {
+ "null", R"("OGC:CRS84")", R"("EPSG:4326")",
+ // Purely the parts of the PROJJSON that we inspect to check the lon/lat case
+ R"({"id": {"authority": "OGC", "code": "CRS84"}})",
+ R"({"id": {"authority": "EPSG", "code": 4326}})"};
+
+ std::string geoarrow_lonlatish_crs = geoarrow_lonlat[0];
Review Comment:
This line is stale, isn't it?
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ const std::string kSridPrefix{"srid:"};
+ const std::string kProjjsonPrefix{"projjson:"};
+
+ // Two reccomendataions are explicitly mentioned in the Parquet format for the
+ // LogicalType crs:
+ //
+ // - "srid:XXXX" as a way to encode an application-specific integer identifier
+ // - "projjson:some_field_name" as a way to avoid repeating PROJJSON strings
+ // unnecessarily (with a suggestion to place them in the file metadata)
+ //
+ // While we don't currently generate those values to reduce the complexity
+ // of the writer, we do interpret these values according to the suggestion in
+ // the format and pass on this information to GeoArrow.
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (::arrow::internal::StartsWith(crs, kSridPrefix)) {
+ return R"("crs": ")" + crs.substr(kSridPrefix.size()) + R"(", "crs_type": "srid")";
+ } else if (::arrow::internal::StartsWith(crs, kProjjsonPrefix)) {
+ std::string metadata_field = crs.substr(kProjjsonPrefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ }
+ }
+
+ // Pass on the string directly to GeoArrow. If the string is already valid JSON,
+ // insert it directly into GeoArrow's "crs" field. Otherwise, escape it and pass it as a
+ // string value.
+ namespace rj = ::arrow::rapidjson;
+ rj::Document document;
+ if (document.Parse(crs.data(), crs.length()).HasParseError()) {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ writer.String(crs);
+ return R"("crs": )" + std::string(buffer.GetString());
+ } else {
+ return R"("crs": )" + crs;
+ }
+}
+
+} // namespace
+
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ std::string_view serialized_data) {
+ // Parquet has no way to interpret a null or missing CRS, so we choose the most likely
+ // intent here (that the user meant to use the default Parquet CRS)
+ if (serialized_data.empty() || serialized_data == "{}") {
+ return LogicalType::Geometry();
+ }
+
+ namespace rj = ::arrow::rapidjson;
+ rj::Document document;
+ if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+ return ::arrow::Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+ }
+
+ ARROW_ASSIGN_OR_RAISE(std::string crs, GeospatialGeoArrowCrsToParquetCrs(document));
+
+ if (document.HasMember("edges") && document["edges"] == "planar") {
+ return LogicalType::Geometry(crs);
+ } else if (document.HasMember("edges") && document["edges"] == "spherical") {
+ return LogicalType::Geography(crs,
+ LogicalType::EdgeInterpolationAlgorithm::SPHERICAL);
+ } else if (document.HasMember("edges")) {
+ return ::arrow::Status::Invalid("Unsupported GeoArrow edge type: ", serialized_data);
+ }
+
+ return LogicalType::Geometry(crs);
+}
+
+::arrow::Result<std::shared_ptr<::arrow::DataType>> GeoArrowTypeFromLogicalType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ // Check if we have a registered GeoArrow type to read into
+ std::shared_ptr<::arrow::ExtensionType> maybe_geoarrow_wkb =
+ ::arrow::GetExtensionType("geoarrow.wkb");
Review Comment:
GeoArrow uses the same extension type for both geometry and geography?
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1343,6 +1390,96 @@ TEST_F(TestConvertArrowSchema, ParquetFlatPrimitivesAsDictionaries) {
ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
}
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsLonLat) {
+ // All the Arrow Schemas below should convert to the type defaults for GEOMETRY
+ // and GEOGRAPHY when GeoArrow extension types are registered and the appropriate
+ // writer option is set.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ // There are several ways that longitude/latitude could be specified when coming from
+ // GeoArrow, which allows null, missing, arbitrary strings (e.g., Authority:Code), and
+ // PROJJSON.
+ std::vector<std::string> geoarrow_lonlat = {
+ "null", R"("OGC:CRS84")", R"("EPSG:4326")",
+ // Purely the parts of the PROJJSON that we inspect to check the lon/lat case
+ R"({"id": {"authority": "OGC", "code": "CRS84"}})",
+ R"({"id": {"authority": "EPSG", "code": 4326}})"};
+
+ std::string geoarrow_lonlatish_crs = geoarrow_lonlat[0];
+ for (const auto& geoarrow_lonlatish_crs : geoarrow_lonlat) {
+ SCOPED_TRACE(geoarrow_lonlatish_crs);
+ std::vector<std::shared_ptr<Field>> arrow_fields = {
+ ::arrow::field("geometry",
+ test::geoarrow_wkb(R"({"crs": )" + geoarrow_lonlatish_crs + "}"),
+ true),
+ ::arrow::field("geography",
+ test::geoarrow_wkb(R"({"crs": )" + geoarrow_lonlatish_crs +
+ R"(, "edges": "spherical"})"),
+ true)};
+
+ ASSERT_OK(ConvertSchema(arrow_fields));
+ ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
+ }
+}
+
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsSrid) {
+ // Checks that it is possible to write the srid:xxxx reccomendation from GeoArrow
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry("srid:1234"),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography("srid:5678"),
+ ParquetType::BYTE_ARRAY));
+
+ std::vector<std::shared_ptr<Field>> arrow_fields = {
+ ::arrow::field("geometry", test::geoarrow_wkb(R"({"crs": "srid:1234"})"), true),
+ ::arrow::field("geography",
+ test::geoarrow_wkb(R"({"crs": "srid:5678", "edges": "spherical"})"),
+ true)};
+
+ ASSERT_OK(ConvertSchema(arrow_fields));
+ ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
+}
+
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsProjjson) {
+ // Checks the conversion between GeoArrow that contains non-lon/lat PROJJSON
+ // to Parquet. Almost all GeoArrow types that arrive at the Parquet reader
+ // will have their CRS expressed in this way.
Review Comment:
```suggestion
// Checks the conversion from GeoArrow that contains non-lon/lat PROJJSON
// to Parquet. Almost all GeoArrow types that arrive at the Parquet reader
// will have their CRS expressed in this way.
```
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
Review Comment:
Would it be hard to implement? It's a bit weird and user-unfriendly to error out here.
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax))) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ geospatial::BoundingBox box;
+
+ if (encoded.xy_bounds_present) {
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+ } else {
+ box.min[0] = kNaN;
+ box.max[0] = kNaN;
+ box.min[1] = kNaN;
+ box.max[1] = kNaN;
+ }
+
+ if (encoded.z_bounds_present) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ } else {
+ box.min[2] = kNaN;
+ box.max[2] = kNaN;
+ }
+
+ if (encoded.m_bounds_present) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ } else {
+ box.min[3] = kNaN;
+ box.max[3] = kNaN;
+ }
+
+ bounder_.MergeBox(box);
+ bounder_.MergeGeometryTypes(encoded.geospatial_types);
+ has_geometry_types_ = has_geometry_types_ && encoded.geospatial_types_present();
+ }
+
+ bool is_wraparound_x() const {
+ return is_wraparound(lower_bound()[0], upper_bound()[0]);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ bool bounds_empty() const {
+ for (int i = 0; i < kMaxDimensions; i++) {
+ if (!bound_empty(i)) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ bool bound_empty(int i) const {
+ return std::isinf(bounder_.Bounds().min[i] - bounder_.Bounds().max[i]);
+ }
+
+ bool bound_valid(int i) const {
+ return !std::isnan(bounder_.Bounds().min[i]) && !std::isnan(bounder_.Bounds().max[i]);
+ }
+
+ const std::array<double, kMaxDimensions>& lower_bound() const {
+ return bounder_.Bounds().min;
+ }
+
+ const std::array<double, kMaxDimensions>& upper_bound() const {
+ return bounder_.Bounds().max;
+ }
+
+ std::optional<std::vector<int32_t>> geometry_types() const {
+ if (has_geometry_types_) {
+ return bounder_.GeometryTypes();
+ } else {
+ return std::nullopt;
+ }
+ }
+
+ private:
+ geospatial::WKBGeometryBounder bounder_;
+ bool is_valid_ = true;
+ bool has_geometry_types_ = true;
+
+ template <typename ArrayType>
+ void UpdateArrayImpl(const ::arrow::Array& values) {
+ const auto& binary_array = static_cast<const ArrayType&>(values);
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
+ if (!binary_array.IsNull(i)) {
+ try {
+ bounder_.MergeGeometry(binary_array.GetView(i));
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+ }
+
+ static bool is_wraparound(double dmin, double dmax) {
Review Comment:
Is this because of the longitude case where the domain is more of a torus? If so, please add a comment explaining this.
##########
cpp/src/parquet/geospatial/statistics_test.cc:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial/statistics.h"
+#include "parquet/test_util.h"
+
+static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+namespace parquet::geospatial {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_TRUE(stats.geometry_types().has_value());
+ EXPECT_EQ(stats.geometry_types().value().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.dimension_empty(), ::testing::ElementsAre(true, true, true, true));
+ EXPECT_THAT(stats.dimension_valid(), ::testing::ElementsAre(true, true, true, true));
+ for (int i = 0; i < kMaxDimensions; i++) {
+ EXPECT_EQ(stats.lower_bound()[i], kInf);
+ EXPECT_EQ(stats.upper_bound()[i], -kInf);
+ }
+
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
Review Comment:
We're not testing `Equals` appropriately if we're not testing cases where it returns false.
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1343,6 +1390,96 @@ TEST_F(TestConvertArrowSchema, ParquetFlatPrimitivesAsDictionaries) {
ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
}
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsLonLat) {
+ // All the Arrow Schemas below should convert to the type defaults for GEOMETRY
+ // and GEOGRAPHY when GeoArrow extension types are registered and the appropriate
+ // writer option is set.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ // There are several ways that longitude/latitude could be specified when coming from
+ // GeoArrow, which allows null, missing, arbitrary strings (e.g., Authority:Code), and
+ // PROJJSON.
+ std::vector<std::string> geoarrow_lonlat = {
+ "null", R"("OGC:CRS84")", R"("EPSG:4326")",
+ // Purely the parts of the PROJJSON that we inspect to check the lon/lat case
+ R"({"id": {"authority": "OGC", "code": "CRS84"}})",
+ R"({"id": {"authority": "EPSG", "code": 4326}})"};
+
+ std::string geoarrow_lonlatish_crs = geoarrow_lonlat[0];
+ for (const auto& geoarrow_lonlatish_crs : geoarrow_lonlat) {
+ SCOPED_TRACE(geoarrow_lonlatish_crs);
Review Comment:
```suggestion
ARROW_SCOPED_TRACE("crs = ", geoarrow_lonlatish_crs);
```
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1343,6 +1390,96 @@ TEST_F(TestConvertArrowSchema, ParquetFlatPrimitivesAsDictionaries) {
ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
}
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsLonLat) {
+ // All the Arrow Schemas below should convert to the type defaults for GEOMETRY
+ // and GEOGRAPHY when GeoArrow extension types are registered and the appropriate
+ // writer option is set.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ // There are several ways that longitude/latitude could be specified when coming from
+ // GeoArrow, which allows null, missing, arbitrary strings (e.g., Authority:Code), and
+ // PROJJSON.
+ std::vector<std::string> geoarrow_lonlat = {
+ "null", R"("OGC:CRS84")", R"("EPSG:4326")",
+ // Purely the parts of the PROJJSON that we inspect to check the lon/lat case
+ R"({"id": {"authority": "OGC", "code": "CRS84"}})",
+ R"({"id": {"authority": "EPSG", "code": 4326}})"};
Review Comment:
Is 4326 always serialized as a number or can it also be serialized as a string?
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +832,60 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+/// \brief Number of bytes in a WKB Point with X and Y dimensions (uint8_t endian,
+/// uint32_t geometry type, 2 * double ordinates)
Review Comment:
```suggestion
/// \brief Number of bytes in a WKB Point with X and Y dimensions (uint8_t endian,
/// uint32_t geometry type, 2 * double coordinates)
```
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -332,6 +408,35 @@ static inline format::SortingColumn ToThrift(SortingColumn sorting_column) {
return thrift_sorting_column;
}
+static inline format::GeospatialStatistics ToThrift(
+ const geospatial::EncodedGeoStatistics& encoded_geo_stats) {
+ format::GeospatialStatistics geospatial_statistics;
+
+ geospatial_statistics.__set_geospatial_types(encoded_geo_stats.geospatial_types);
+
+ format::BoundingBox bbox;
+ if (encoded_geo_stats.xy_bounds_present) {
+ bbox.__set_xmin(encoded_geo_stats.xmin);
+ bbox.__set_xmax(encoded_geo_stats.xmax);
+ bbox.__set_ymin(encoded_geo_stats.ymin);
+ bbox.__set_ymax(encoded_geo_stats.ymax);
+
+ if (encoded_geo_stats.z_bounds_present) {
+ bbox.__set_zmin(encoded_geo_stats.zmin);
+ bbox.__set_zmax(encoded_geo_stats.zmax);
+ }
+
+ if (encoded_geo_stats.m_bounds_present) {
+ bbox.__set_mmin(encoded_geo_stats.mmin);
+ bbox.__set_mmax(encoded_geo_stats.mmax);
+ }
+
+ geospatial_statistics.__set_bbox(bbox);
Review Comment:
```suggestion
geospatial_statistics.__set_bbox(std::move(bbox));
```
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -232,6 +233,81 @@ static inline AadMetadata FromThrift(format::AesGcmCtrV1 aesGcmCtrV1) {
aesGcmCtrV1.supply_aad_prefix};
}
+static inline geospatial::EncodedGeoStatistics FromThrift(
+ const format::GeospatialStatistics& geo_stats) {
+ geospatial::EncodedGeoStatistics out;
+
+ out.geospatial_types = geo_stats.geospatial_types;
+
+ if (geo_stats.__isset.bbox) {
+ out.xmin = geo_stats.bbox.xmin;
+ out.xmax = geo_stats.bbox.xmax;
+ out.ymin = geo_stats.bbox.ymin;
+ out.ymax = geo_stats.bbox.ymax;
+ out.xy_bounds_present = true;
+
+ if (geo_stats.bbox.__isset.zmin && geo_stats.bbox.__isset.zmax) {
+ out.zmin = geo_stats.bbox.zmin;
+ out.zmax = geo_stats.bbox.zmax;
+ out.z_bounds_present = true;
+ } else {
+ out.z_bounds_present = false;
Review Comment:
I don't think the `else` branch is required since the various "present" fields are initialized to false?
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -332,6 +408,35 @@ static inline format::SortingColumn ToThrift(SortingColumn sorting_column) {
return thrift_sorting_column;
}
+static inline format::GeospatialStatistics ToThrift(
+ const geospatial::EncodedGeoStatistics& encoded_geo_stats) {
+ format::GeospatialStatistics geospatial_statistics;
+
+ geospatial_statistics.__set_geospatial_types(encoded_geo_stats.geospatial_types);
+
+ format::BoundingBox bbox;
+ if (encoded_geo_stats.xy_bounds_present) {
Review Comment:
Can move the `bbox` instantiation inside the `if`.
```suggestion
if (encoded_geo_stats.xy_bounds_present) {
format::BoundingBox bbox;
```
##########
cpp/src/parquet/types.cc:
##########
@@ -1630,6 +1669,217 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+namespace {
+std::string EscapeControl(char c) {
+ std::stringstream ss;
+ ss << R"(\u00)";
+ ss.flags(ss.hex);
+ ss.width(2);
+ ss.fill('0');
+ ss << static_cast<int>(c);
+ return ss.str();
+}
+
+void WriteCrsKeyAndValue(const std::string& crs, std::ostream& json) {
+ // There is no restriction on the crs value here, and it may contain quotes
+ // or backslashes that would result in invalid JSON if unescaped.
+ json << R"(, "crs": ")";
+ json.flags(json.hex);
+ for (char c : crs) {
+ if (c == '"') {
+ json << R"(\")";
+ } else if (c == '\\') {
+ json << R"(\\)";
+ } else if (c >= 0 && c < 32) {
+ json << EscapeControl(c);
+ } else {
+ json << c;
+ }
+ }
+ json << R"(")";
+}
+} // namespace
+
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ WriteCrsKeyAndValue(crs_, json);
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto logical_type = std::shared_ptr<GeometryLogicalType>(new GeometryLogicalType());
Review Comment:
`std::make_shared` is more idiomatic and also saves an allocation
```suggestion
auto logical_type = std::make_shared<GeometryLogicalType>();
```
##########
cpp/src/parquet/types.cc:
##########
@@ -1630,6 +1669,217 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+namespace {
+std::string EscapeControl(char c) {
Review Comment:
You're writing the JSON string into a stream anyway, so why not do that here as well?
```suggestion
void EscapeControl(char c, std::ostream& json) {
```
##########
cpp/src/parquet/types.cc:
##########
@@ -479,6 +480,27 @@ std::shared_ptr<const LogicalType> LogicalType::FromThrift(
return UUIDLogicalType::Make();
} else if (type.__isset.FLOAT16) {
return Float16LogicalType::Make();
+ } else if (type.__isset.GEOMETRY) {
+ std::string crs;
+ if (type.GEOMETRY.__isset.crs) {
+ crs = type.GEOMETRY.crs;
+ }
+
+ return GeometryLogicalType::Make(crs);
Review Comment:
```suggestion
return GeometryLogicalType::Make(std::move(crs));
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,136 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types (experimental)"""
+
+ def __init__(self):
+ raise TypeError(f"Do not call {self.__class__.__name__}'s constructor directly")
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ cdef optional[vector[int32_t]] maybe_geometry_types = \
+ self.statistics.get().geometry_types()
+ if not maybe_geometry_types.has_value():
+ return None
+
+ return list(maybe_geometry_types.value())
+
+ @property
+ def lower_bound(self):
+ return [self.statistics.get().lower_bound()[i] for i in range(4)]
+
+ @property
+ def upper_bound(self):
+ return [self.statistics.get().upper_bound()[i] for i in range(4)]
+
+ @property
+ def dimension_empty(self):
+ return [self.statistics.get().dimension_empty()[i] for i in range(4)]
+
+ @property
+ def dimension_valid(self):
+ return [self.statistics.get().dimension_valid()[i] for i in range(4)]
+
+ @property
+ def has_x(self):
+ return self.dimension_valid[0] and not self.dimension_empty[0]
+
+ @property
+ def has_y(self):
+ return self.dimension_valid[1] and not self.dimension_empty[1]
+
+ @property
+ def has_z(self):
+ return self.dimension_valid[2] and not self.dimension_empty[2]
+
+ @property
+ def has_m(self):
+ return self.dimension_valid[3] and not self.dimension_empty[3]
+
+ @property
+ def xmin(self):
+ if self.has_x:
+ return self.lower_bound[0]
+ else:
+ return None
Review Comment:
For the record, you can make these snippets shorter if you want:
```suggestion
return self.lower_bound[0] if self.has_x else None
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,136 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types (experimental)"""
+
+ def __init__(self):
+ raise TypeError(f"Do not call {self.__class__.__name__}'s constructor directly")
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ cdef optional[vector[int32_t]] maybe_geometry_types = \
+ self.statistics.get().geometry_types()
+ if not maybe_geometry_types.has_value():
+ return None
+
+ return list(maybe_geometry_types.value())
+
+ @property
+ def lower_bound(self):
+ return [self.statistics.get().lower_bound()[i] for i in range(4)]
+
+ @property
+ def upper_bound(self):
+ return [self.statistics.get().upper_bound()[i] for i in range(4)]
+
+ @property
+ def dimension_empty(self):
+ return [self.statistics.get().dimension_empty()[i] for i in range(4)]
+
+ @property
+ def dimension_valid(self):
+ return [self.statistics.get().dimension_valid()[i] for i in range(4)]
+
+ @property
+ def has_x(self):
+ return self.dimension_valid[0] and not self.dimension_empty[0]
+
+ @property
+ def has_y(self):
+ return self.dimension_valid[1] and not self.dimension_empty[1]
+
+ @property
+ def has_z(self):
+ return self.dimension_valid[2] and not self.dimension_empty[2]
+
+ @property
+ def has_m(self):
+ return self.dimension_valid[3] and not self.dimension_empty[3]
+
+ @property
+ def xmin(self):
+ if self.has_x:
+ return self.lower_bound[0]
+ else:
+ return None
Review Comment:
And by the way, this will call the `lower_bound` property every time that `xmin`, `ymin`, etc. is called. Not very efficient, so perhaps you want something like:
```suggestion
return self.statistics.get().lower_bound()[0] if self.has_x else None
```
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
+
+ private:
+ GeographyLogicalType() = default;
Review Comment:
Hmm, this may be meant to prevent direct instantiation. Let's keep it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953931206
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
Review Comment:
nit: please document non trivial functions. A better name for Func might be `Consume` or `CoordConsumer consumer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1956565641
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ if (!is_valid_ || !other.is_valid_) {
+ is_valid_ = false;
+ return;
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ buf.Init(item.ptr, item.len);
+ ::arrow::Status status = bounder_.ReadGeometry(&buf);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ buf.Init(item.ptr, item.len);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(&buf));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ const auto& binary_array = static_cast<const ::arrow::BinaryArray&>(values);
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
Review Comment:
I feel like there's something I can do with SFINAE here and something more efficient than checking is_null at the element level (but it also looks like binary views are rejected by the writer, so I left it at binary + large binary for now). It's true that the level of branching incurred in the bounder is unlikely to make that matter much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1957011653
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,8 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geometry_statistics.cc
Review Comment:
We have renamed `GeometryStatistics` to `GeospatialStatistics` in the Parquet spec to avoid confusion that it does not support geography type. Perhaps we should rename the file and class names too.
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto* logical_type = new GeometryLogicalType();
+ logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
+ return std::shared_ptr<const LogicalType>(logical_type);
+}
+
+class LogicalType::Impl::Geography final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeographyLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const {
+ return algorithm_;
+ }
+
+ const char* algorithm_name() const {
Review Comment:
```suggestion
std::string_view algorithm_name() const {
```
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -232,6 +233,29 @@ static inline AadMetadata FromThrift(format::AesGcmCtrV1 aesGcmCtrV1) {
aesGcmCtrV1.supply_aad_prefix};
}
+static inline EncodedGeospatialStatistics FromThrift(
+ const format::GeospatialStatistics& geometry_stats) {
Review Comment:
```suggestion
const format::GeospatialStatistics& geospatial_stats) {
```
or `geo_stats` for short?
##########
cpp/src/parquet/metadata.h:
##########
@@ -31,6 +31,20 @@
namespace parquet {
+class ColumnDescriptor;
Review Comment:
Please use `parquet/type_fwd.h` and add missing ones there.
##########
cpp/src/parquet/metadata.h:
##########
@@ -141,8 +155,10 @@ class PARQUET_EXPORT ColumnChunkMetaData {
int64_t num_values() const;
std::shared_ptr<schema::ColumnPath> path_in_schema() const;
bool is_stats_set() const;
+ bool is_geometry_stats_set() const;
std::shared_ptr<Statistics> statistics() const;
std::shared_ptr<SizeStatistics> size_statistics() const;
+ std::shared_ptr<GeospatialStatistics> geometry_statistics() const;
Review Comment:
```suggestion
std::shared_ptr<GeospatialStatistics> geospatial_statistics() const;
```
##########
cpp/src/parquet/metadata.cc:
##########
@@ -111,6 +112,17 @@ static std::shared_ptr<Statistics> MakeTypedColumnStats(
metadata.statistics.__isset.null_count, metadata.statistics.__isset.distinct_count);
}
+static std::shared_ptr<GeospatialStatistics> MakeColumnGeometryStats(
Review Comment:
```suggestion
static std::shared_ptr<GeospatialStatistics> MakeColumnGeospatialStats(
```
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
+ return R"("crs": ")" + crs.substr(srid_prefix.size()) + R"(", "crs_type": "srid")";
+ } else if (crs.rfind(projjson_prefix, 0) == 0) {
+ std::string metadata_field = crs.substr(projjson_prefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ } else {
+ // Pass on the value of the field so the user can sort this out if needed
+ return R"("crs": )" + metadata_field + R"(, "crs_type": "projjson")";
+ }
+ } else {
+ return Status::Invalid("Can't convert invalid Parquet CRS string to GeoArrow: ", crs);
Review Comment:
Does it mean that we should enforce srid/projjson prefix from the `crs` string if we want to use GeoArrow extension type? If true, we need to document this expectation somewhere.
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1113,12 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// Write GEOMETRY and GEOGRAPHY logical types where possible.
Review Comment:
Why do we need this?
##########
cpp/src/parquet/types.cc:
##########
@@ -479,6 +479,38 @@ std::shared_ptr<const LogicalType> LogicalType::FromThrift(
return UUIDLogicalType::Make();
} else if (type.__isset.FLOAT16) {
return Float16LogicalType::Make();
+ } else if (type.__isset.GEOMETRY) {
+ std::string crs;
+ if (type.GEOMETRY.__isset.crs) {
+ crs = type.GEOMETRY.crs;
+ }
+
+ return GeometryLogicalType::Make(crs);
+ } else if (type.__isset.GEOGRAPHY) {
+ std::string crs;
+ if (type.GEOGRAPHY.__isset.crs) {
+ crs = type.GEOGRAPHY.crs;
+ }
+
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ LogicalType::EdgeInterpolationAlgorithm::UNKNOWN;
+ if (!type.GEOGRAPHY.__isset.algorithm ||
+ type.GEOGRAPHY.algorithm == format::EdgeInterpolationAlgorithm::SPHERICAL) {
+ algorithm = LogicalType::EdgeInterpolationAlgorithm::SPHERICAL;
Review Comment:
nit: replace these lines by defining `FromThrift` and `ToThrift` functions in the `thrift_internal.h`
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
Review Comment:
```suggestion
const std::string kSridPrefix{"srid:"};
const std::string kProjjsonPrefix{"projjson:"};
```
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
+ return R"("crs": ")" + crs.substr(srid_prefix.size()) + R"(", "crs_type": "srid")";
+ } else if (crs.rfind(projjson_prefix, 0) == 0) {
+ std::string metadata_field = crs.substr(projjson_prefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ } else {
+ // Pass on the value of the field so the user can sort this out if needed
+ return R"("crs": )" + metadata_field + R"(, "crs_type": "projjson")";
+ }
+ } else {
+ return Status::Invalid("Can't convert invalid Parquet CRS string to GeoArrow: ", crs);
+ }
+}
+
+Result<std::shared_ptr<ArrowType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ // Check if we have a registered GeoArrow type to read into
+ std::shared_ptr<::arrow::ExtensionType> maybe_geoarrow_wkb =
+ ::arrow::GetExtensionType("geoarrow.wkb");
+ if (!maybe_geoarrow_wkb) {
+ return ::arrow::binary();
+ }
+
+ if (logical_type.is_geometry()) {
+ const auto& geospatial_type = checked_cast<const GeometryLogicalType&>(logical_type);
+ ARROW_ASSIGN_OR_RAISE(std::string crs_metadata,
+ MakeGeoArrowCrsMetadata(geospatial_type.crs(), metadata));
+
+ std::string serialized_data = std::string("{") + crs_metadata + "}";
+ return maybe_geoarrow_wkb->Deserialize(::arrow::binary(), serialized_data);
Review Comment:
CMIW, the GeoArrow extension is responsible for deserializing the json metadata?
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -20,6 +20,7 @@
#include "arrow/extension/json.h"
#include "arrow/type.h"
+#include "arrow/util/key_value_metadata.h"
#include "parquet/properties.h"
Review Comment:
```suggestion
#include "arrow/extension/json.h"
#include "arrow/type.h"
#include "arrow/util/key_value_metadata.h"
#include "parquet/properties.h"
```
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
Review Comment:
Why do we need this? BTW it is better to use std::string_view IMHO.
Perhaps we can remove it by providing `std::string_view EdgeInterpolationAlgorithmToString(EdgeInterpolationAlgorithm t)` similar to other `XXXToString` functions below in this file?
##########
cpp/src/parquet/metadata.h:
##########
@@ -141,8 +155,10 @@ class PARQUET_EXPORT ColumnChunkMetaData {
int64_t num_values() const;
std::shared_ptr<schema::ColumnPath> path_in_schema() const;
bool is_stats_set() const;
+ bool is_geometry_stats_set() const;
Review Comment:
```suggestion
bool is_geo_stats_set() const;
```
##########
cpp/src/parquet/metadata.h:
##########
@@ -440,6 +456,9 @@ class PARQUET_EXPORT ColumnChunkMetaDataBuilder {
void SetStatistics(const EncodedStatistics& stats);
void SetSizeStatistics(const SizeStatistics& size_stats);
+ // column geometry statistics
+ void SetGeospatialStatistics(const EncodedGeospatialStatistics& geometry_stats);
Review Comment:
```suggestion
void SetGeospatialStatistics(const EncodedGeospatialStatistics& geo_stats);
```
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
+ return R"("crs": ")" + crs.substr(srid_prefix.size()) + R"(", "crs_type": "srid")";
+ } else if (crs.rfind(projjson_prefix, 0) == 0) {
+ std::string metadata_field = crs.substr(projjson_prefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ } else {
+ // Pass on the value of the field so the user can sort this out if needed
+ return R"("crs": )" + metadata_field + R"(, "crs_type": "projjson")";
+ }
+ } else {
+ return Status::Invalid("Can't convert invalid Parquet CRS string to GeoArrow: ", crs);
+ }
+}
+
+Result<std::shared_ptr<ArrowType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ // Check if we have a registered GeoArrow type to read into
+ std::shared_ptr<::arrow::ExtensionType> maybe_geoarrow_wkb =
+ ::arrow::GetExtensionType("geoarrow.wkb");
+ if (!maybe_geoarrow_wkb) {
+ return ::arrow::binary();
+ }
+
+ if (logical_type.is_geometry()) {
+ const auto& geospatial_type = checked_cast<const GeometryLogicalType&>(logical_type);
+ ARROW_ASSIGN_OR_RAISE(std::string crs_metadata,
+ MakeGeoArrowCrsMetadata(geospatial_type.crs(), metadata));
+
+ std::string serialized_data = std::string("{") + crs_metadata + "}";
+ return maybe_geoarrow_wkb->Deserialize(::arrow::binary(), serialized_data);
+ } else {
Review Comment:
Should we check `logical_type.is_geography()` here and throw in the else branch?
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +322,15 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geometry_stats_set() const {
+ DCHECK(writer_version_ != nullptr);
Review Comment:
```suggestion
```
We don't need this actually.
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -332,6 +356,27 @@ static inline format::SortingColumn ToThrift(SortingColumn sorting_column) {
return thrift_sorting_column;
}
+static inline format::GeospatialStatistics ToThrift(
+ const EncodedGeospatialStatistics& encoded_geometry_stats) {
Review Comment:
ditto
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto* logical_type = new GeometryLogicalType();
+ logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
+ return std::shared_ptr<const LogicalType>(logical_type);
+}
+
+class LogicalType::Impl::Geography final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeographyLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const {
+ return algorithm_;
+ }
+
+ const char* algorithm_name() const {
+ switch (algorithm_) {
+ case LogicalType::EdgeInterpolationAlgorithm::SPHERICAL:
+ return "spherical";
+ case LogicalType::EdgeInterpolationAlgorithm::VINCENTY:
+ return "vincenty";
+ case LogicalType::EdgeInterpolationAlgorithm::THOMAS:
+ return "thomas";
+ case LogicalType::EdgeInterpolationAlgorithm::ANDOYER:
+ return "andoyer";
+ case LogicalType::EdgeInterpolationAlgorithm::KARNEY:
+ return "karney";
+ default:
+ return "unknown";
+ }
+ }
+
+ private:
+ Geography(std::string crs, LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm)
+ : LogicalType::Impl(LogicalType::Type::GEOGRAPHY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)),
+ algorithm_(algorithm) {}
+
+ std::string crs_;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm_;
+};
+
+std::string LogicalType::Impl::Geography::ToString() const {
+ std::stringstream type;
+ type << "Geography(crs=" << crs_ << ", algorithm=" << algorithm_name() << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geography::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geography")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ if (algorithm_ != LogicalType::EdgeInterpolationAlgorithm::SPHERICAL) {
+ json << R"(, "algorithm": ")" << algorithm_name() << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geography::ToThrift() const {
+ format::LogicalType type;
+ format::GeographyType geography_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geography_type.__set_crs(crs_);
+ }
+
+ if (algorithm_ == LogicalType::EdgeInterpolationAlgorithm::SPHERICAL) {
Review Comment:
ditto
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +322,15 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geometry_stats_set() const {
+ DCHECK(writer_version_ != nullptr);
Review Comment:
Perhaps we even don't need `is_geometry_stats_set()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974547075
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
Review Comment:
I added an `is_empty()` check.
> BTW, I think C++ and Java implementations should be consistent
I agree, although I am also trying to be consistent with `Statistics` (maybe that is not important, though). I am hesitant to expose *another* bounding box class in a public header (many libraries already have one and it's very close to the statistics class) but I certainly could.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974521159
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -428,13 +520,20 @@ Status FieldToNode(const std::string& name, const std::shared_ptr<Field>& field,
}
case ArrowTypeId::EXTENSION: {
auto ext_type = std::static_pointer_cast<::arrow::ExtensionType>(field->type());
- // Built-in JSON extension is handled differently.
+ // Built-in JSON extension and GeoArrow are handled differently.
if (ext_type->extension_name() == std::string("arrow.json")) {
// Set physical and logical types and instantiate primitive node.
type = ParquetType::BYTE_ARRAY;
logical_type = LogicalType::JSON();
break;
+ } else if (arrow_properties.write_geospatial_logical_types() &&
Review Comment:
I'll write a top-level comment below about this one...I basically kept it for now but moved everything to a place where the JSON dependency can be removed (and I'm totally happy to remove the JSON part and tackle getting this to a place where it's actually useful in a separate PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974504661
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,8 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geometry_statistics.cc
Review Comment:
I think I caught all of these! (Shortening `Geospatial` to `Geo` where possible according to another suggestion, although I'm happy to change the names of any of those 🙂 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975880837
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
Review Comment:
I added parameter names here (I don't remember why they started this way!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975815368
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
+ }
+
+ ASSERT_OK(builder.AppendNull());
+
+ for (int k = 10; k < 15; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
+ }
+
+ // Ensure we have both a binary array and a large binary array to work with
+ ASSERT_OK_AND_ASSIGN(const auto binary_array, builder.Finish());
+ ASSERT_OK_AND_ASSIGN(const auto large_binary_array,
+ ::arrow::compute::Cast(binary_array, ::arrow::large_binary()));
+
+ GeospatialStatistics stats;
+ stats.Update(*binary_array);
+ EXPECT_EQ(stats.GetXMin(), 0);
+ EXPECT_EQ(stats.GetYMin(), 1);
+ EXPECT_EQ(stats.GetXMax(), 14);
+ EXPECT_EQ(stats.GetYMax(), 15);
Review Comment:
We could...this one is more about whether or not `Update()` is getting called correctly (the tests above it check whether or not x, y, z, and m are all piped through the `GeoStatistics` and `EncodedGeoStatistics`). If it's surprising I'm happy to update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975820496
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
Review Comment:
I converted this from exceptions to `Status` at the request of an earlier review...I don't personally mind which idiom it uses but I left it as `Status` for now and could change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975891793
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
Review Comment:
I think I successfully got everything renamed to `Geospatial` or `Geo`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975909959
##########
cpp/src/parquet/types.h:
##########
@@ -166,6 +168,17 @@ class PARQUET_EXPORT LogicalType {
enum unit { UNKNOWN = 0, MILLIS = 1, MICROS, NANOS };
};
+ struct EdgeInterpolationAlgorithm {
+ enum algorithm {
Review Comment:
I changed this to an enum class (although I think it looks a little funny next to in the header `TimeUnit::unit`). It may be worth updating this for all of these enums at some point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974508831
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -20,6 +20,7 @@
#include "arrow/extension/json.h"
#include "arrow/type.h"
+#include "arrow/util/key_value_metadata.h"
#include "parquet/properties.h"
Review Comment:
I think I got everything alphabetized in all the headers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008350733
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
Review Comment:
Previous code reused the buffer (but no longer does, so this is removed!)
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+namespace {
+
+::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMin);
+ auto max_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMax);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMin);
+ auto max_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMax);
+
+ if (geometry_type_component < min_geometry_type_value ||
+ geometry_type_component > max_geometry_type_value ||
+ dimensions_component < min_dimension_value ||
+ dimensions_component > max_dimension_value) {
+ return ::arrow::Status::SerializationError("Invalid WKB geometry type: ",
+ wkb_geometry_type);
+ }
+
+ GeometryTypeAndDimensions out{static_cast<GeometryType>(geometry_type_component),
+ static_cast<Dimensions>(dimensions_component)};
+ return out;
+}
+
+} // namespace
+
+std::vector<int32_t> WKBGeometryBounder::GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometry(std::string_view bytes_wkb) {
+ WKBBuffer src{reinterpret_cast<const uint8_t*>(bytes_wkb.data()),
+ static_cast<int64_t>(bytes_wkb.size())};
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(&src, /*record_wkb_type=*/true));
+ if (src.size() != 0) {
+ return ::arrow::Status::SerializationError(
+ "Exepcted zero bytes after consuming WKB but got ", src.size());
+ }
+
+ return ::arrow::Status::OK();
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometryInternal(WKBBuffer* src,
+ bool record_wkb_type) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007948931
##########
cpp/src/parquet/geospatial_util_internal_json.h:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/properties.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromGeoArrowJSON(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties);
+
+/// \brief Compute a suitable DataType into which a GEOMETRY or GEOGRAPHY type should be
+/// read
+///
+/// The result of this function depends on whether or not "geoarrow.wkb" has been
+/// registered: if it has, the result will be the registered ExtensionType; if it has not,
+/// the result will be binary().
+::arrow::Result<std::shared_ptr<::arrow::DataType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata);
+
+/// \brief The default GeoCrsContext, which writes any PROJJSON coordinate reference
+/// system values it encounters to the file metadata
+class PARQUET_EXPORT FileGeoCrsContext : public GeoCrsContext {
Review Comment:
I removed the GeoCrsContext from this PR (I'll write a note about this below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007976819
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970051391
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
Review Comment:
Yes, `GeometryType` is definitely a less confusing option (but here we are 🙂 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1971173784
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
Review Comment:
I don't remember the precise API name, but it certainly exists :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016684442
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
I'd rather not make it mandatory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057982516
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
I would suggest this diff:
```diff
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 1af347914f..92ea7e346f 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -374,16 +374,13 @@ target_include_directories(arrow::flatbuffers
# ----------------------------------------------------------------------
# Some EP's require other EP's
-if(PARQUET_REQUIRE_ENCRYPTION)
- set(ARROW_JSON ON)
-endif()
-
if(ARROW_WITH_OPENTELEMETRY)
set(ARROW_WITH_NLOHMANN_JSON ON)
set(ARROW_WITH_PROTOBUF ON)
endif()
if(ARROW_PARQUET)
+ set(ARROW_WITH_RAPIDJSON ON)
set(ARROW_WITH_THRIFT ON)
endif()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059536882
##########
cpp/src/parquet/geospatial/util_internal.h:
##########
@@ -0,0 +1,216 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <array>
+#include <cmath>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging_internal.h"
+#include "parquet/platform.h"
+
+namespace parquet::geospatial {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct PARQUET_EXPORT BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const;
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Throws ParquetException for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ void MergeGeometry(std::string_view bytes_wkb);
Review Comment:
Done! (And used in the tests to make them more concise!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2836231939
Thank you all for the reviews!
Further comments welcome, or if not, I'll merge tomorrow afternoon and start on some general Parquet follow-up work 🙂 (removing the minimal dependency bit from CMake, adding null statistics when sort order is unknown).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2066297645
##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -0,0 +1,238 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_internal.h"
+
+#include <sstream>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geospatial {
+
+std::string BoundingBox::ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+}
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ uint8_t ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ uint32_t ReadUInt32(bool swap) {
+ auto value = ReadChecked<uint32_t>();
+ if (swap) {
+ return ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ void ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ throw ParquetException("Can't read coordinate sequence of ", total_bytes,
+ " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (swap) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (auto& c : coord) {
+ c = ::arrow::bit_util::ByteSwap(c);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
Review Comment:
Well, you could use the [`subspan`](https://en.cppreference.com/w/cpp/container/span/subspan) method. Anyway, this is not really important, so you don't need to do this if it doesn't make the code clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2025007053
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status MergeGeometry(std::string_view bytes_wkb);
Review Comment:
Agreed with @wgtmac . If we want to keep the `Status` return then we can put them inside `arrow/util` or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2050744275
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,393 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = !bound_empty(2) && bound_valid(2);
+ bool write_m = !bound_empty(3) && bound_valid(3);
Review Comment:
I think functionally that happened (but it was tucked away in the thrift_internal.h). Your version is clearer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017867509
##########
cpp/src/parquet/geospatial_statistics_test.cc:
##########
@@ -0,0 +1,317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_EQ(stats.geometry_types().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_TRUE(stats.is_empty());
+ EXPECT_FALSE(stats.has_x());
+ EXPECT_FALSE(stats.has_y());
+ EXPECT_FALSE(stats.has_z());
+ EXPECT_FALSE(stats.has_m());
+ EXPECT_EQ(stats.xmax() - stats.xmin(), -kInf);
+ EXPECT_EQ(stats.ymax() - stats.ymin(), -kInf);
+ EXPECT_EQ(stats.zmax() - stats.zmin(), -kInf);
+ EXPECT_EQ(stats.mmax() - stats.mmin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [inf, -inf]\n y: [inf, -inf]\n geometry_types:\n");
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_x());
+ EXPECT_FALSE(encoded.has_y());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
Review Comment:
I added a few more checks here!
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
Review Comment:
Done (it did!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016134264
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read coordinate sequence of ", total_bytes,
+ " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
Review Comment:
```suggestion
for (auto& c: coord) {
c = ByteSwap(c);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016117185
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
Review Comment:
Can you put this in the anonymous namespace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016157968
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ return list(self.statistics.geospatial_types)
Review Comment:
This is going to return a list of integers?
Perhaps call this `raw_geospatial_types` in case we want to later return a list of strings or Python enum values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Updated) Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2641949811
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1963826327
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
Review Comment:
Are you looking for `StartsWith` perhaps?
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1102,9 +1105,18 @@ int64_t ColumnWriterImpl::Close() {
if (rows_written_ > 0 && chunk_statistics.is_set()) {
metadata_->SetStatistics(chunk_statistics);
}
+
if (rows_written_ > 0 && chunk_size_statistics.is_set()) {
metadata_->SetSizeStatistics(chunk_size_statistics);
}
+
+ if (descr_->logical_type() != nullptr && descr_->logical_type()->is_geometry()) {
+ EncodedGeospatialStatistics geometry_stats = GetChunkGeospatialStatistics();
+ if (geometry_stats.is_set()) {
+ metadata_->SetGeospatialStatistics(geometry_stats);
Review Comment:
Can probably move the structure?
```suggestion
metadata_->SetGeospatialStatistics(std::move(geometry_stats));
```
##########
cpp/src/parquet/column_writer_test.cc:
##########
@@ -399,6 +399,26 @@ class TestPrimitiveWriter : public PrimitiveTypedTest<TestType> {
return metadata_accessor->key_value_metadata();
}
+ std::unique_ptr<ColumnChunkMetaData> metadata_accessor() {
+ // Metadata accessor must be created lazily.
+ // This is because the ColumnChunkMetaData semantics dictate the metadata object is
+ // complete (no changes to the metadata buffer can be made after instantiation)
+ return ColumnChunkMetaData::Make(metadata_->contents(), this->descr_);
+ }
+
+ EncodedStatistics metadata_encoded_stats() { return metadata_stats()->Encode(); }
+
+ std::shared_ptr<Statistics> metadata_stats() {
+ return metadata_accessor()->statistics();
+ }
+
+ std::shared_ptr<GeospatialStatistics> metadata_geometry_stats() {
+ ApplicationVersion app_version(this->writer_properties_->created_by());
+ auto metadata_accessor = ColumnChunkMetaData::Make(
+ metadata_->contents(), this->descr_, default_reader_properties(), &app_version);
Review Comment:
Why is this not using the same code as in `metadata_accessor()` above?
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status = bounder_.ReadGeometry(item.ptr, item.len);
+ if (!status.ok()) {
Review Comment:
Why wouldn't we throw an exception here instead?
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -21,6 +21,12 @@
#include <string>
#include <vector>
+// TODO(paleolimbot): Remove once example files are generated
Review Comment:
When will that be done?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
Review Comment:
Why these? The compiler should probably synthesize them implicitly, no?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
Review Comment:
It says "update" but the method is called "Decode", so what does it do actually?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
Review Comment:
What is an invalid WKB and why wouldn't it throw an error instead of silently produce empty stats?
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
Review Comment:
Why does `stats.Update` take a pointer? It could take a `ByteArray` value as it's cheap to copy.
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
Review Comment:
Since it has a `Equals` method it can also probably inherit from `EqualityComparable` in order to get (in)equality operators.
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
Review Comment:
There is a strong risk of short uppercase names like this to clash with public macros from third-party headers (we already know that MIN and MAX are problematic).
Can we find more distinctive names? Perhaps at least `kXY`, `kXYZ`, etc.
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
Review Comment:
`ByteArray` has a constructor that accepts a `string_view`, so you can perhaps write `ByteArray item0(xyzm_wkb0)`, or at worse `ByteArray item0(std::string_view(xyzm_wkb0))`
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
Review Comment:
Why is this `clone` method needed?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
Review Comment:
You don't need this constructor to be public, do you?
(actually, you perhaps don't need it at all)
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
Review Comment:
Are these useful? They should be synthesized implicitly by the compiler, no?
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status = bounder_.ReadGeometry(item.ptr, item.len);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(item.ptr, item.len));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException(
+ "Unsupported Array type in GeospatialStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeospatialStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geometry::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geometry::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeospatialStatistics out;
+ out.geospatial_types = bounder_.GeometryTypes();
+
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.zmin = mins[2];
Review Comment:
Do we always emit z and m? In `Update` below we handle the case where they don't exist.
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1396,6 +1412,12 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
return result;
}
+ EncodedGeospatialStatistics GetChunkGeospatialStatistics() override {
+ EncodedGeospatialStatistics result;
+ if (chunk_geometry_statistics_) result = chunk_geometry_statistics_->Encode();
+ return result;
Review Comment:
Or perhaps shorter
```suggestion
return chunk_geometry_statistics_ ? chunk_geometry_statistics_->Encode() : {};
```
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
Review Comment:
I don't think you need to define this one explicitly.
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
Review Comment:
Why 3001? Do you want to comment here?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
Review Comment:
Why force the user to pass a `std::array` for each of these methods? A `span<const double>` would probably be more flexible.
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
Review Comment:
Please remove spurious default constructor decls where possible.
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
Review Comment:
Why `z`? this is confusing...
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
Review Comment:
Can this perhaps take a `span<const int32_t>` instead?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
Review Comment:
`checked_cast` perhaps?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
Review Comment:
What is `true` for?
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
+ uint32_t wkb_type =
+ static_cast<int>(item.dimensions) * 1000 + static_cast<int>(item.geometry_type);
+ EXPECT_THAT(bounder.GeometryTypes(), ::testing::ElementsAre(::testing::Eq(wkb_type)));
+
+ bounder.Reset();
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+ EXPECT_TRUE(bounder.GeometryTypes().empty());
+}
+
+INSTANTIATE_TEST_SUITE_P(
+ TestGeometryUtil, WKBTestFixture,
+ ::testing::Values(
+ // POINT (30 10)
+ WKBTestCase(GeometryType::POINT, Dimensions::XY,
+ {0x01, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // POINT Z (30 10 40)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZ,
+ {0x01, 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // POINT M (30 10 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYM,
+ {0x01, 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // POINT ZM (30 10 40 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZM,
+ {0x01, 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // LINESTRING (30 10, 10 30, 40 40)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XY,
+ {0x01, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // LINESTRING Z (30 10 40, 10 30 40, 40 40 80)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZ,
+ {0x01, 0xea, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // LINESTRING M (30 10 300, 10 30 300, 40 40 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYM,
+ {0x01, 0xd2, 0x07, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0,
+ 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // LINESTRING ZM (30 10 40 300, 10 30 40 300, 40 40 80 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZM,
+ {0x01, 0xba, 0x0b, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
+ WKBTestCase(GeometryType::POLYGON, Dimensions::XY,
+ {0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZ,
+ {0x01, 0xeb, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // POLYGON M ((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYM,
+ {0x01, 0xd3, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // POLYGON ZM ((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30 10 40
+ // 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZM,
+ {0x01, 0xbb, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // MULTIPOINT ((30 10))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XY,
+ {0x01, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // MULTIPOINT Z ((30 10 40))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZ,
+ {0x01, 0xec, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // MULTIPOINT M ((30 10 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYM,
+ {0x01, 0xd4, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // MULTIPOINT ZM ((30 10 40 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZM,
+ {0x01, 0xbc, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // MULTILINESTRING ((30 10, 10 30, 40 40))
+ WKBTestCase(GeometryType::MULTILINESTRING, Dimensions::XY,
+ {0x01, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x02,
+ 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // MULTILINESTRING Z ((30 10 40, 10 30 40, 40 40 80))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZ,
+ {0x01, 0xed, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xea, 0x03, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // MULTILINESTRING M ((30 10 300, 10 30 300, 40 40 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYM,
+ {0x01, 0xd5, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd2, 0x07, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // MULTILINESTRING ZM ((30 10 40 300, 10 30 40 300, 40 40 80 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZM,
+ {0x01, 0xbd, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xba, 0x0b, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XY,
+ {0x01, 0x06, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // MULTIPOLYGON Z (((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYZ,
+ {0x01, 0xee, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xeb, 0x03, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // MULTIPOLYGON M (((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYM,
+ {0x01, 0xd6, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd3, 0x07, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // MULTIPOLYGON ZM (((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30
+ // 10 40 300)))
+ WKBTestCase(GeometryType::MULTIPOLYGON, Dimensions::XYZM,
+ {0x01, 0xbe, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xbb,
+ 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // GEOMETRYCOLLECTION (POINT (30 10))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XY,
+ {0x01, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // GEOMETRYCOLLECTION Z (POINT Z (30 10 40))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZ,
+ {0x01, 0xef, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // GEOMETRYCOLLECTION M (POINT M (30 10 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYM,
+ {0x01, 0xd7, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // GEOMETRYCOLLECTION ZM (POINT ZM (30 10 40 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZM,
+ {0x01, 0xbf, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300})));
+
+struct MakeWKBPointTestCase {
+ MakeWKBPointTestCase() = default;
+ MakeWKBPointTestCase(const std::vector<double> xyzm, bool has_z, bool has_m)
+ : has_z(has_z), has_m(has_m) {
+ memcpy(this->xyzm, xyzm.data(), sizeof(this->xyzm));
+ }
Review Comment:
Are these constructors actually useful, or can you rely on the default-generated constructors?
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
Review Comment:
Could use `EXPECT_EQ` if `operator==` is defined.
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
+ uint32_t wkb_type =
+ static_cast<int>(item.dimensions) * 1000 + static_cast<int>(item.geometry_type);
+ EXPECT_THAT(bounder.GeometryTypes(), ::testing::ElementsAre(::testing::Eq(wkb_type)));
+
+ bounder.Reset();
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+ EXPECT_TRUE(bounder.GeometryTypes().empty());
+}
+
+INSTANTIATE_TEST_SUITE_P(
+ TestGeometryUtil, WKBTestFixture,
+ ::testing::Values(
+ // POINT (30 10)
+ WKBTestCase(GeometryType::POINT, Dimensions::XY,
+ {0x01, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // POINT Z (30 10 40)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZ,
+ {0x01, 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // POINT M (30 10 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYM,
+ {0x01, 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // POINT ZM (30 10 40 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZM,
+ {0x01, 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // LINESTRING (30 10, 10 30, 40 40)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XY,
+ {0x01, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // LINESTRING Z (30 10 40, 10 30 40, 40 40 80)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZ,
+ {0x01, 0xea, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // LINESTRING M (30 10 300, 10 30 300, 40 40 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYM,
+ {0x01, 0xd2, 0x07, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0,
+ 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // LINESTRING ZM (30 10 40 300, 10 30 40 300, 40 40 80 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZM,
+ {0x01, 0xba, 0x0b, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
+ WKBTestCase(GeometryType::POLYGON, Dimensions::XY,
+ {0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZ,
+ {0x01, 0xeb, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // POLYGON M ((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYM,
+ {0x01, 0xd3, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // POLYGON ZM ((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30 10 40
+ // 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZM,
+ {0x01, 0xbb, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // MULTIPOINT ((30 10))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XY,
+ {0x01, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // MULTIPOINT Z ((30 10 40))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZ,
+ {0x01, 0xec, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // MULTIPOINT M ((30 10 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYM,
+ {0x01, 0xd4, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // MULTIPOINT ZM ((30 10 40 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZM,
+ {0x01, 0xbc, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // MULTILINESTRING ((30 10, 10 30, 40 40))
+ WKBTestCase(GeometryType::MULTILINESTRING, Dimensions::XY,
+ {0x01, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x02,
+ 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // MULTILINESTRING Z ((30 10 40, 10 30 40, 40 40 80))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZ,
+ {0x01, 0xed, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xea, 0x03, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // MULTILINESTRING M ((30 10 300, 10 30 300, 40 40 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYM,
+ {0x01, 0xd5, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd2, 0x07, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // MULTILINESTRING ZM ((30 10 40 300, 10 30 40 300, 40 40 80 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZM,
+ {0x01, 0xbd, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xba, 0x0b, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XY,
+ {0x01, 0x06, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // MULTIPOLYGON Z (((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYZ,
+ {0x01, 0xee, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xeb, 0x03, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // MULTIPOLYGON M (((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYM,
+ {0x01, 0xd6, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd3, 0x07, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // MULTIPOLYGON ZM (((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30
+ // 10 40 300)))
+ WKBTestCase(GeometryType::MULTIPOLYGON, Dimensions::XYZM,
+ {0x01, 0xbe, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xbb,
+ 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // GEOMETRYCOLLECTION (POINT (30 10))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XY,
+ {0x01, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // GEOMETRYCOLLECTION Z (POINT Z (30 10 40))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZ,
+ {0x01, 0xef, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // GEOMETRYCOLLECTION M (POINT M (30 10 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYM,
+ {0x01, 0xd7, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // GEOMETRYCOLLECTION ZM (POINT ZM (30 10 40 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZM,
+ {0x01, 0xbf, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300})));
+
+struct MakeWKBPointTestCase {
+ MakeWKBPointTestCase() = default;
+ MakeWKBPointTestCase(const std::vector<double> xyzm, bool has_z, bool has_m)
+ : has_z(has_z), has_m(has_m) {
+ memcpy(this->xyzm, xyzm.data(), sizeof(this->xyzm));
+ }
+
+ double xyzm[4];
Review Comment:
Can we use std::array here?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetXMax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
+ /// case, these bounds represent the union of the intervals [ymax, Inf] and [-Inf,
+ /// ymin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetYMin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no Y values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetYMax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMax() const;
+
+ /// \brief Returns true if any Z values were encountered or false otherwise
+ bool HasZ() const;
+
+ /// \brief Returns true if any M values were encountered or false otherwise
+ bool HasM() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> GetGeometryTypes() const;
+
+ private:
Review Comment:
Can we add a `ToString` method to aid in debugging?
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
+ }
+
+ ASSERT_OK(builder.AppendNull());
+
+ for (int k = 10; k < 15; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
+ }
+
+ // Ensure we have both a binary array and a large binary array to work with
+ ASSERT_OK_AND_ASSIGN(const auto binary_array, builder.Finish());
+ ASSERT_OK_AND_ASSIGN(const auto large_binary_array,
+ ::arrow::compute::Cast(binary_array, ::arrow::large_binary()));
+
+ GeospatialStatistics stats;
+ stats.Update(*binary_array);
+ EXPECT_EQ(stats.GetXMin(), 0);
+ EXPECT_EQ(stats.GetYMin(), 1);
+ EXPECT_EQ(stats.GetXMax(), 14);
+ EXPECT_EQ(stats.GetYMax(), 15);
Review Comment:
Don't we want to test Z and M bounds as well?
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
Review Comment:
We have a lot of such snippets, perhaps we could write something like
```suggestion
EXPECT_EQ(stats.GetBounds(), {10, 20, 11, 21, 12, 22, 13, 23});
```
or perhaps using a test-only helper (if you don't think this deserves an API)
```suggestion
EXPECT_EQ(GetBounds(stats), {10, 20, 11, 21, 12, 22, 13, 23});
```
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
Review Comment:
Let's use `kPoint`, etc.
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
Review Comment:
Why not check `ToString` here too?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
Review Comment:
Are MIN and MAX needed?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
Review Comment:
Can we use a consistent naming? (either "geometry" or "geospatial")
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
Review Comment:
Can probably use a string_view?
```suggestion
ASSERT_OK(builder.AppendValue(std::string_view(item.data(), item.size()));
```
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
Review Comment:
It might (or might not!) be more convenient to have something like:
```c++
/// If coords.size() == 2, return XY
/// If coords.size() == 3 and has_z == true, return XYZ
/// If coords.size() == 3 and has_z == false, return XYM
/// If coords.size() == 4, return XYZM
std::string MakeWKBPoint(span<const double> coords, bool has_z = true)
```
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
Review Comment:
Remove?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
Review Comment:
What's the point of `enable_write_page_index` if we're never checking page index contents?
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
Review Comment:
Perhaps check idempotence here?
```suggestion
EXPECT_EQ(bounder.Bounds(), item.box);
ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
EXPECT_EQ(bounder.Bounds(), item.box);
```
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
Review Comment:
These functions don't need to be inlined in `test_util.h` IMO, can they be defined in `test_util.cc` instead?
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes = 21 + (has_z ? 8 : 0) + (has_m ? 8 : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::POINT, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
+
+ return wkb;
+}
+
+static constexpr int kWkbPointSize = 21; // 1:endianness + 4:type + 8:x + 8:y
+
+inline void GenerateWKBPoint(uint8_t* ptr, double x, double y) {
+ double xyzm[] = {x, y, geometry::kInf, geometry::kInf};
+ std::string wkb = MakeWKBPoint(xyzm, false, false);
+ std::memcpy(ptr, wkb.data(), kWkbPointSize);
Review Comment:
Are we sure `wkb` will be of size `kWkbPointSize`?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ double x = 0;
+ double y = 0;
+ EXPECT_TRUE(test::GetWKBPointCoordinate(value, &x, &y));
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_DOUBLE_EQ(expected_x, x);
+ EXPECT_DOUBLE_EQ(expected_y, y);
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeospatialStatistics(std::shared_ptr<GeospatialStatistics> geom_stats) {
+ ASSERT_TRUE(geom_stats != nullptr);
+ std::vector<int32_t> geospatial_types = geom_stats->GetGeometryTypes();
+ EXPECT_EQ(1, geospatial_types.size());
+ EXPECT_EQ(1, geospatial_types[0]);
Review Comment:
What does "1" correspond to? Add a comment perhaps?
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
Review Comment:
Perhaps it would be nicer to immediately see the omitted dimensions. For example (but feel free to find a better convention):
```suggestion
EXPECT_EQ(box.ToString(),
"BoundingBox [x: (), y: (), z: (), m: ()]");
```
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
Review Comment:
In non-Arrow Parquet APIs, we probably want to throw exceptions, not return `Status`.
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
Review Comment:
Why sort? Is this always useful or only in some cases? What is the typical size of the output here?
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ double x = 0;
+ double y = 0;
+ EXPECT_TRUE(test::GetWKBPointCoordinate(value, &x, &y));
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_DOUBLE_EQ(expected_x, x);
+ EXPECT_DOUBLE_EQ(expected_y, y);
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeospatialStatistics(std::shared_ptr<GeospatialStatistics> geom_stats) {
+ ASSERT_TRUE(geom_stats != nullptr);
+ std::vector<int32_t> geospatial_types = geom_stats->GetGeometryTypes();
+ EXPECT_EQ(1, geospatial_types.size());
+ EXPECT_EQ(1, geospatial_types[0]);
+ EXPECT_GE(geom_stats->GetXMin(), 0);
+ EXPECT_GT(geom_stats->GetXMax(), geom_stats->GetXMin());
+ EXPECT_GT(geom_stats->GetYMin(), 0);
+ EXPECT_GT(geom_stats->GetYMax(), geom_stats->GetYMin());
Review Comment:
How come these are all zero?
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes = 21 + (has_z ? 8 : 0) + (has_m ? 8 : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::POINT, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
+
+ return wkb;
+}
+
+static constexpr int kWkbPointSize = 21; // 1:endianness + 4:type + 8:x + 8:y
+
+inline void GenerateWKBPoint(uint8_t* ptr, double x, double y) {
+ double xyzm[] = {x, y, geometry::kInf, geometry::kInf};
+ std::string wkb = MakeWKBPoint(xyzm, false, false);
+ std::memcpy(ptr, wkb.data(), kWkbPointSize);
+}
+
+inline bool GetWKBPointCoordinate(const ByteArray& value, double* out_x, double* out_y) {
Review Comment:
Perhaps this can be
```suggestion
std::optional<std::pair<double>> GetWKBPointCoordinate(const ByteArray& value);
```
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ /// \brief Reset the internal bounds and geometry types list to an empty state
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ ::arrow::Status ReadGeometryInternal(WKBBuffer* src, bool record_wkb_type);
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions dimensions, uint32_t n_coords,
+ bool swap);
Review Comment:
Are these used anywhere? I see they are `private`.
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
Review Comment:
Well, we can have both, as `EdgeInterpolationAlgorithmToString` is gonna be tedious to find, write and read.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -312,6 +312,93 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeospatialStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return """{}
+ geospatial_types: {}
+ xmin: {}
+ xmax: {}
+ ymin: {}
Review Comment:
Not a very pretty output... especially the multi-line aspect.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -312,6 +312,93 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeospatialStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return """{}
Review Comment:
You can use a f-string, which will be quite easier and more readable IMHO.
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -332,6 +356,27 @@ static inline format::SortingColumn ToThrift(SortingColumn sorting_column) {
return thrift_sorting_column;
}
+static inline format::GeospatialStatistics ToThrift(
+ const EncodedGeospatialStatistics& encoded_geometry_stats) {
+ format::GeospatialStatistics geometry_statistics;
Review Comment:
`geo_stats` here?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -440,6 +537,20 @@ cdef class ColumnChunkMetaData(_Weakrefable):
statistics.init(self.metadata.statistics(), self)
return statistics
+ @property
+ def is_geometry_stats_set(self):
+ """Whether or not geometry statistics are present in metadata (bool)."""
+ return self.metadata.is_geometry_stats_set()
+
+ @property
+ def geospatial_statistics(self):
Review Comment:
Geometry? Geospatial?
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
+
+ private:
+ GeographyLogicalType() = default;
Review Comment:
Useful?
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
Review Comment:
`checked_cast` perhaps?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
Review Comment:
Also can this take a `std::string_view` (or, alternatively, a `span<const uint8_t>`)?
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1113,12 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// Write GEOMETRY and GEOGRAPHY logical types where possible.
Review Comment:
Right, is there a use case for disabling this?
##########
cpp/src/parquet/types.h:
##########
@@ -166,6 +168,17 @@ class PARQUET_EXPORT LogicalType {
enum unit { UNKNOWN = 0, MILLIS = 1, MICROS, NANOS };
};
+ struct EdgeInterpolationAlgorithm {
+ enum algorithm {
Review Comment:
This is an outdated C++ idiom, we should use a `enum class` instead:
```c++
enum class EdgeInterpolationAlgorithm {
UNKNOWN = 0,
```
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -232,6 +233,29 @@ static inline AadMetadata FromThrift(format::AesGcmCtrV1 aesGcmCtrV1) {
aesGcmCtrV1.supply_aad_prefix};
}
+static inline EncodedGeospatialStatistics FromThrift(
+ const format::GeospatialStatistics& geometry_stats) {
Review Comment:
+1 for adopting the short `geo_stats` convention. Will be much easier.
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
Review Comment:
Also, what are 1 and 0? Perhaps add the parameter names to make this code easier to read.
```suggestion
stats.Update(&item0, /*xxx=*/1, /*yyy=*/0);
```
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetXMax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
+ /// case, these bounds represent the union of the intervals [ymax, Inf] and [-Inf,
+ /// ymin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetYMin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no Y values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetYMax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMax() const;
+
+ /// \brief Returns true if any Z values were encountered or false otherwise
+ bool HasZ() const;
+
+ /// \brief Returns true if any M values were encountered or false otherwise
+ bool HasM() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> GetGeometryTypes() const;
Review Comment:
And why return a copy of `std::vector<int32_t>`? Is it generated on the fly?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
Review Comment:
And, since sorting is heavy-weight, you should move the definition to the `.cc` file. We would like to avoid recompiling this function for every calling module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2655542105
:warning: GitHub issue #45522 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059536007
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Returns false if a bound was explicitly not calculated by a writer when
+ /// importing statistics from thrift
+ ///
+ /// When calculating statistics, this will always be true because this implementation
+ /// calculates statistics for all dimensions.
+ std::array<bool, kMaxDimensions> dimension_valid() const;
Review Comment:
I reworded these docstrings (I went with "not available from the Parquet file", which is the case that dimension_valid() currently communicates). This is conceptually similar to HasMinMax() in the Statistics except here we have to differentiate between "there were no values for a particular dimension" and "was not available from the imported statistics", since this class represents both of those cases. I hope the reworded comments are more helpful 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059543202
##########
cpp/src/parquet/types.cc:
##########
@@ -1630,6 +1669,217 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+namespace {
+std::string EscapeControl(char c) {
Review Comment:
Since we have RapidJSON as required now, I updated this to use RapidJSON to sanitize the CRS as a string.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,136 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types (experimental)"""
+
+ def __init__(self):
+ raise TypeError(f"Do not call {self.__class__.__name__}'s constructor directly")
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2050765717
##########
cpp/src/generated/parquet_types.h:
##########
@@ -4,3888 +4,3888 @@
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
* @generated
*/
-#ifndef parquet_TYPES_H
-#define parquet_TYPES_H
+ #ifndef parquet_TYPES_H
Review Comment:
I accidentally clang-formatted it 😬 (fixed!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969531706
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1396,6 +1412,12 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
return result;
}
+ EncodedGeospatialStatistics GetChunkGeospatialStatistics() override {
+ EncodedGeospatialStatistics result;
+ if (chunk_geometry_statistics_) result = chunk_geometry_statistics_->Encode();
+ return result;
Review Comment:
Done!
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1102,9 +1105,18 @@ int64_t ColumnWriterImpl::Close() {
if (rows_written_ > 0 && chunk_statistics.is_set()) {
metadata_->SetStatistics(chunk_statistics);
}
+
if (rows_written_ > 0 && chunk_size_statistics.is_set()) {
metadata_->SetSizeStatistics(chunk_size_statistics);
}
+
+ if (descr_->logical_type() != nullptr && descr_->logical_type()->is_geometry()) {
+ EncodedGeospatialStatistics geometry_stats = GetChunkGeospatialStatistics();
+ if (geometry_stats.is_set()) {
+ metadata_->SetGeospatialStatistics(geometry_stats);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016075521
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -259,6 +262,12 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # TODO(paleolimbot): Make sure this is OK or remove if not!
+ if(ARROW_JSON)
Review Comment:
I think it's ok to require JSON here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016097558
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
+
+ /// \brief Returns true if any non-NaN Y values were encountered or false otherwise
+ bool has_y() const;
+
+ /// \brief Returns true if any non-NaN Z values were encountered or false otherwise
+ bool has_z() const;
+
+ /// \brief Returns true if any non-NaN M values were encountered or false otherwise
+ bool has_m() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> geometry_types() const;
Review Comment:
Is there a reason this returns a copy and not a const-ref to the vector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016103426
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
Review Comment:
(it can also make the tests more concise :))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016108550
##########
cpp/src/parquet/geospatial_statistics_test.cc:
##########
@@ -0,0 +1,317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_EQ(stats.geometry_types().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_TRUE(stats.is_empty());
+ EXPECT_FALSE(stats.has_x());
+ EXPECT_FALSE(stats.has_y());
+ EXPECT_FALSE(stats.has_z());
+ EXPECT_FALSE(stats.has_m());
+ EXPECT_EQ(stats.xmax() - stats.xmin(), -kInf);
+ EXPECT_EQ(stats.ymax() - stats.ymin(), -kInf);
+ EXPECT_EQ(stats.zmax() - stats.zmin(), -kInf);
+ EXPECT_EQ(stats.mmax() - stats.mmin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [inf, -inf]\n y: [inf, -inf]\n geometry_types:\n");
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_x());
+ EXPECT_FALSE(encoded.has_y());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
Review Comment:
Can we test `Equals` being false sometimes? Otherwise an incomplete implementation of `Equals` would pass the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016139450
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
Review Comment:
Serialized using what? Arrow IPC, Thrift? This not very clear to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016141637
##########
cpp/src/parquet/metadata.cc:
##########
@@ -111,6 +112,16 @@ static std::shared_ptr<Statistics> MakeTypedColumnStats(
metadata.statistics.__isset.null_count, metadata.statistics.__isset.distinct_count);
}
+static std::shared_ptr<GeoStatistics> MakeColumnGeometryStats(
+ const format::ColumnMetaData& metadata, const ColumnDescriptor* descr) {
+ if (metadata.__isset.geospatial_statistics) {
+ EncodedGeoStatistics encoded_geo_stats = FromThrift(metadata.geospatial_statistics);
+ return std::make_shared<GeoStatistics>(encoded_geo_stats);
Review Comment:
```suggestion
return std::make_shared<GeoStatistics>(std::move(encoded_geo_stats));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016173692
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,160 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
Review Comment:
Would it be useful to also test the geography logical type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007924014
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +321,14 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geo_stats_set() const {
+ if (possible_geo_stats_ == nullptr &&
+ column_metadata_->__isset.geospatial_statistics) {
+ possible_geo_stats_ = MakeColumnGeometryStats(*column_metadata_, descr_);
Review Comment:
I don't really mind either way, but the `Statistics` are currently initialized in exactly the same way:
```c
if (possible_stats_ == nullptr) {
possible_stats_ = MakeColumnStats(*column_metadata_, descr_);
}
```
...and so even if I update this there is still no thread safety for the `ColumnChunkMetadata`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008348987
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double get_xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
Review Comment:
I added explicit documentation and tests for how this implementation handles NaNs (and improved the handling of "imported" NaNs if somebody else happened to write them to the thrift file). Basically, we ignore NaNs we see in coordinates (like GEOS, but not like JTS, which propagates NaNs, and not like PostGIS, which resets bounds after the first NaN it sees); however, we ensure that "imported" NaNs propagate (so that when predicate pushdown is implemented a caller can detect this case). I'm open to suggestions on improving that...it's sort of open season among spatial implementations on what might happen if you put an NaN in a non-point geometry.
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975816413
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
Review Comment:
I gave them better names so that perhaps the utility is apparent. They were added at the request of an earlier review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975823035
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
Review Comment:
I think I successfully moved all non-trivial things to `geometry_util_internal.cc`!
The typical size of the output here is one or two elements (e.g., usually people are writing 100% points to a file, even though the WKB could contain any type of geometry). I quite like this as canonically sorted whenever possible (for comparison, either programatically or visually) but it's not strictly necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975891271
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
Review Comment:
I added a `/*leaf_field_nullable*/` comment here...it seems to be required by that method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974548552
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
Review Comment:
I updated this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974517707
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -232,6 +233,29 @@ static inline AadMetadata FromThrift(format::AesGcmCtrV1 aesGcmCtrV1) {
aesGcmCtrV1.supply_aad_prefix};
}
+static inline EncodedGeospatialStatistics FromThrift(
+ const format::GeospatialStatistics& geometry_stats) {
Review Comment:
I think I got most `geospatial` converted to `geo` and it does read much more nicely 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1958667969
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
Review Comment:
I moved the WKBBuffer and the bounder method implementation into an implementation file!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007971208
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1658,193 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto* logical_type = new GeometryLogicalType();
+ logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
+ return std::shared_ptr<const LogicalType>(logical_type);
Review Comment:
I changed this (and the Geography) constructor to match your change!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007967892
##########
cpp/src/parquet/arrow/writer.cc:
##########
@@ -546,6 +554,10 @@ Status GetSchemaMetadata(const ::arrow::Schema& schema, ::arrow::MemoryPool* poo
std::string schema_as_string = serialized->ToString();
std::string schema_base64 = ::arrow::util::base64_encode(schema_as_string);
result->Append(kArrowSchemaKey, std::move(schema_base64));
+
+ // Add any required fields collected by the GeoCrsContext
+ properties.geo_crs_context()->AddProjjsonCrsFieldsToFileMetadata(result.get());
Review Comment:
Agreed! (I removed the GeoCrsContext for now and perhaps a dedicated change could weigh better options for doing this, or perhaps writers will find the workaround of putting `projjson:xxx` and adding the `xxx` to the their own field metadata sufficient).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017867141
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ const std::string& serialized_data);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017865043
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ return list(self.statistics.geospatial_types)
Review Comment:
I'm hesitant to depart from the names/definitions in the Parquet format Thrift too much but I don't feel all that strongly about it if you think this would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016080105
##########
cpp/src/parquet/api/reader.h:
##########
@@ -22,6 +22,7 @@
#include "parquet/column_scanner.h"
#include "parquet/exception.h"
#include "parquet/file_reader.h"
+#include "parquet/geospatial_statistics.h"
Review Comment:
Is this required/desirable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017995991
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
I'm not sure exactly how to proceed with this one...it's easy enough to remove the `if(ARROW_JSON)` from the CMake lists and remove these...are there also follow-ups that are a reasonable scope for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017853950
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,9 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geospatial_statistics.cc
Review Comment:
The `parquet/` directory doesn't currently use subdirectories for other files with related functionality (e.g., there are more `level_` files than `geospatial_` ones), although I'm happy to move them anywhere!
Perhaps a bigger question, but I hope that there are not any more geo data utilities coming to Parquet (if anything, I'd like to figure out how to move some of these things out of Parquet using a canonical extension type or compute functions registered at runtime).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016144230
##########
cpp/src/parquet/test_util.h:
##########
@@ -39,6 +39,7 @@
#include "parquet/column_reader.h"
#include "parquet/column_writer.h"
#include "parquet/encoding.h"
+#include "parquet/geospatial_util_internal.h"
Review Comment:
Are these additional includes required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017488860
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status MergeGeometry(std::string_view bytes_wkb);
Review Comment:
@emkornfield suggested this early on and I updated this to use the Status instead of ParquetException. I don't have any opinions about which one it uses (this is internal and easy to change) but I do happen to like Status better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017496308
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1104,9 +1107,18 @@ int64_t ColumnWriterImpl::Close() {
if (rows_written_ > 0 && chunk_statistics.is_set()) {
metadata_->SetStatistics(chunk_statistics);
}
+
if (rows_written_ > 0 && chunk_size_statistics.is_set()) {
metadata_->SetSizeStatistics(chunk_size_statistics);
}
+
+ if (descr_->logical_type() != nullptr && descr_->logical_type()->is_geometry()) {
Review Comment:
Geography columns can have statistics but generating them is a battle for a different day 🙂 (it's much more complicated than a simple min/max by dimension and probably requires figuring out how to wire up an existing implementation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974551072
##########
cpp/src/parquet/geometry_util_internal.cc:
##########
@@ -0,0 +1,234 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
Review Comment:
I went with `string_view` at Antoine's suggestion (but could add an overload with something else if needed!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974507931
##########
cpp/src/parquet/types.cc:
##########
@@ -479,6 +479,38 @@ std::shared_ptr<const LogicalType> LogicalType::FromThrift(
return UUIDLogicalType::Make();
} else if (type.__isset.FLOAT16) {
return Float16LogicalType::Make();
+ } else if (type.__isset.GEOMETRY) {
+ std::string crs;
+ if (type.GEOMETRY.__isset.crs) {
+ crs = type.GEOMETRY.crs;
+ }
+
+ return GeometryLogicalType::Make(crs);
+ } else if (type.__isset.GEOGRAPHY) {
+ std::string crs;
+ if (type.GEOGRAPHY.__isset.crs) {
+ crs = type.GEOGRAPHY.crs;
+ }
+
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ LogicalType::EdgeInterpolationAlgorithm::UNKNOWN;
+ if (!type.GEOGRAPHY.__isset.algorithm ||
+ type.GEOGRAPHY.algorithm == format::EdgeInterpolationAlgorithm::SPHERICAL) {
+ algorithm = LogicalType::EdgeInterpolationAlgorithm::SPHERICAL;
Review Comment:
I moved these both to `thrift_internal.h`!
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto* logical_type = new GeometryLogicalType();
+ logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
+ return std::shared_ptr<const LogicalType>(logical_type);
+}
+
+class LogicalType::Impl::Geography final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeographyLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const {
+ return algorithm_;
+ }
+
+ const char* algorithm_name() const {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975793191
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
Review Comment:
It was used for `clone()`, which I removed! (And this too!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975902982
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
Review Comment:
I think I successfully removed anything from test_util.h that wasn't used in a a test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975800987
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
Review Comment:
I'll put this in my top level comment below shortly, but I could be convinced either way on this one. I don't think we validate JSON or UUIDs either, but also we don't generate statistics for them that involve actually parsing them. There are some other flavours of WKB that some libraries emit (e.g., EWKB from PostGIS or GEOS before a certain version), and some geometry types that are sometimes written that we don't support (curved things, TINs, etc.) which would sort of be like JSON with comments being put in a JSON column. Is that error material or would it end up just being annoying for a user? I'm honestly not sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975806723
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
Review Comment:
I updated all of these and it's much cleaner! (`ByteArray item0(xyzm_wkb0)` seemed to work.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975886996
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
+ uint32_t wkb_type =
+ static_cast<int>(item.dimensions) * 1000 + static_cast<int>(item.geometry_type);
+ EXPECT_THAT(bounder.GeometryTypes(), ::testing::ElementsAre(::testing::Eq(wkb_type)));
+
+ bounder.Reset();
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+ EXPECT_TRUE(bounder.GeometryTypes().empty());
+}
+
+INSTANTIATE_TEST_SUITE_P(
+ TestGeometryUtil, WKBTestFixture,
+ ::testing::Values(
+ // POINT (30 10)
+ WKBTestCase(GeometryType::POINT, Dimensions::XY,
+ {0x01, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // POINT Z (30 10 40)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZ,
+ {0x01, 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // POINT M (30 10 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYM,
+ {0x01, 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // POINT ZM (30 10 40 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZM,
+ {0x01, 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // LINESTRING (30 10, 10 30, 40 40)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XY,
+ {0x01, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // LINESTRING Z (30 10 40, 10 30 40, 40 40 80)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZ,
+ {0x01, 0xea, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // LINESTRING M (30 10 300, 10 30 300, 40 40 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYM,
+ {0x01, 0xd2, 0x07, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0,
+ 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // LINESTRING ZM (30 10 40 300, 10 30 40 300, 40 40 80 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZM,
+ {0x01, 0xba, 0x0b, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
+ WKBTestCase(GeometryType::POLYGON, Dimensions::XY,
+ {0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZ,
+ {0x01, 0xeb, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // POLYGON M ((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYM,
+ {0x01, 0xd3, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // POLYGON ZM ((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30 10 40
+ // 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZM,
+ {0x01, 0xbb, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // MULTIPOINT ((30 10))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XY,
+ {0x01, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // MULTIPOINT Z ((30 10 40))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZ,
+ {0x01, 0xec, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // MULTIPOINT M ((30 10 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYM,
+ {0x01, 0xd4, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // MULTIPOINT ZM ((30 10 40 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZM,
+ {0x01, 0xbc, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // MULTILINESTRING ((30 10, 10 30, 40 40))
+ WKBTestCase(GeometryType::MULTILINESTRING, Dimensions::XY,
+ {0x01, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x02,
+ 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // MULTILINESTRING Z ((30 10 40, 10 30 40, 40 40 80))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZ,
+ {0x01, 0xed, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xea, 0x03, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // MULTILINESTRING M ((30 10 300, 10 30 300, 40 40 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYM,
+ {0x01, 0xd5, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd2, 0x07, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // MULTILINESTRING ZM ((30 10 40 300, 10 30 40 300, 40 40 80 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZM,
+ {0x01, 0xbd, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xba, 0x0b, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XY,
+ {0x01, 0x06, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // MULTIPOLYGON Z (((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYZ,
+ {0x01, 0xee, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xeb, 0x03, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // MULTIPOLYGON M (((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYM,
+ {0x01, 0xd6, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd3, 0x07, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // MULTIPOLYGON ZM (((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30
+ // 10 40 300)))
+ WKBTestCase(GeometryType::MULTIPOLYGON, Dimensions::XYZM,
+ {0x01, 0xbe, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xbb,
+ 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // GEOMETRYCOLLECTION (POINT (30 10))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XY,
+ {0x01, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // GEOMETRYCOLLECTION Z (POINT Z (30 10 40))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZ,
+ {0x01, 0xef, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // GEOMETRYCOLLECTION M (POINT M (30 10 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYM,
+ {0x01, 0xd7, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // GEOMETRYCOLLECTION ZM (POINT ZM (30 10 40 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZM,
+ {0x01, 0xbf, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300})));
+
+struct MakeWKBPointTestCase {
+ MakeWKBPointTestCase() = default;
+ MakeWKBPointTestCase(const std::vector<double> xyzm, bool has_z, bool has_m)
+ : has_z(has_z), has_m(has_m) {
+ memcpy(this->xyzm, xyzm.data(), sizeof(this->xyzm));
+ }
Review Comment:
I removed them!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2050775500
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool writer_calculated_xy_bounds{};
+ double xmin{};
+ double xmax{};
+ double ymin{};
+ double ymax{};
+
+ bool writer_calculated_z_bounds{};
Review Comment:
I went with `z_bound_present`, since that's what it's communicating to (or from) thrift. When this is decoded by the GeoStatistics, we then check the x and y bound for nan-ness and represent invalid (i.e., had nan) and "not present" the same. Happy to workshop a better name for those concepts!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2024526762
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
+
+ /// \brief Returns true if any non-NaN Y values were encountered or false otherwise
+ bool has_y() const;
+
+ /// \brief Returns true if any non-NaN Z values were encountered or false otherwise
+ bool has_z() const;
+
+ /// \brief Returns true if any non-NaN M values were encountered or false otherwise
+ bool has_m() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> geometry_types() const;
Review Comment:
Ok, if the vector is short I guess it's not much of a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2733455619
Sorry for the delay! I was busy with my employer in the past weeks. I will take a look this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1961812496
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -428,13 +520,20 @@ Status FieldToNode(const std::string& name, const std::shared_ptr<Field>& field,
}
case ArrowTypeId::EXTENSION: {
auto ext_type = std::static_pointer_cast<::arrow::ExtensionType>(field->type());
- // Built-in JSON extension is handled differently.
+ // Built-in JSON extension and GeoArrow are handled differently.
if (ext_type->extension_name() == std::string("arrow.json")) {
// Set physical and logical types and instantiate primitive node.
type = ParquetType::BYTE_ARRAY;
logical_type = LogicalType::JSON();
break;
+ } else if (arrow_properties.write_geospatial_logical_types() &&
Review Comment:
Will you eventually remove GeoArrow from here? The current implementation can only write geometry/geography types from GeoArrow extension types. I also have concern if parquet-cpp will depend on rapidjson.
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1792,6 +1831,12 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
page_statistics_->IncrementNullCount(num_chunk_levels - non_null_count);
page_statistics_->IncrementNumValues(non_null_count);
page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false);
+
+ if constexpr (std::is_same<T, ByteArray>::value) {
+ if (chunk_geometry_statistics_ != nullptr) {
+ chunk_geometry_statistics_->Update(*referenced_dictionary);
Review Comment:
I'm skeptical that we need to support dictionary-encoded geometry data.
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
Review Comment:
Add a comment `experimental`?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetXMax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
+ /// case, these bounds represent the union of the intervals [ymax, Inf] and [-Inf,
+ /// ymin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetYMin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no Y values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetYMax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMax() const;
+
+ /// \brief Returns true if any Z values were encountered or false otherwise
+ bool HasZ() const;
+
+ /// \brief Returns true if any M values were encountered or false otherwise
+ bool HasM() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> GetGeometryTypes() const;
Review Comment:
It seems that these `GetXXX` and `HasXXX` functions are trivial and should use snake case?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
Review Comment:
Is it better to use `std::optional<double>` instead of returning Inf?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
Review Comment:
Should we add has_x() and has_y()? What if all values are null so the EncodedGeospatialStatistics is empty? What if x or y has any NaN?
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
Review Comment:
> while writing a file
Reading a file also can return the collected stats so I think the docstring is not accurate.
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
Review Comment:
```suggestion
std::unique_ptr<GeospatialStatistics> clone() const;
```
Make it more flexible.
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
Review Comment:
Shouldn't we need to check min/max?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2672158509
@jorisvandenbossche I think this is right up your alley, do you want to help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2673110425
@pitrou @wgtmac Thank you for the detailed reviews! I am working through these now but wanted to share my high-level plan just to make sure I'm headed in the right direction. Basically, I'd like to remove as much as possible from this PR while still allowing a route to read/write files that with GEOMETRY and GEOGRAPHY types.
- Create a separate branch for writing example files with arbitrary CRSes ( https://github.com/apache/arrow/compare/main...paleolimbot:arrow:parquet-geo-write-files-from-geoarrow ). It's important to generate those files but the scope of this PR should shrink to a more reasonable scope based on your collective comments. Some of those files could also be used in tests here (or in a future PR)...the testing for some other types seems to use this mechanism.
- Remove GeoArrow CRS support when writing Parquet (and thus remove the RapidJSON dependency). In this case we'd still be able to provide a mechanism for writing basically the "default CRS" case. I'll remove the `write_geospatial_extensions` writer option (it seems like we are in agreement on this although I don't feel strongly either way)
- Keep support for reading into GeoArrow. This doesn't require a JSON parser and would still let us read arbitrary files (when opted in via `allow_arrow_extensions`. Most of this would ideally live in an extension type but removing it would make it difficult to test the PR. It would be opt-in, so would be difficult to accidentally depend on (it could be also marked experimental and could be changed pending the results of a proper mailing list discussion).
- Raise the issue of `geoarrow.wkb` as a canonical extension type. I've spoken with Joris and Kyle about this but need to get their consensus on how to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1987871488
##########
cpp/src/parquet/geospatial_util_internal_json.cc:
##########
Review Comment:
@jorisvandenbossche This is the file that has the metadata conversion between the Parquet and GeoArrow we just chatted about!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955161427
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
Review Comment:
This was designed to mimic how enums are defined in `types.h` (e.g., `TimeUnit::unit`), but I agree that a normal `enum` is way better. I removed the functions that weren't essential and moved FromWKB into the WKB bounder where it's more clear what it's doing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955189344
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
Review Comment:
I added versions of these! (I went with `ReadXXX` but I'm not particularly attached 🙂 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1958669003
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) >= 0; }
+
+ bool has_m() const { return (mmax - mmin) >= 0; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+class PARQUET_EXPORT GeospatialStatistics {
Review Comment:
I added documentation here leaning toward the verbose side of things (but happy to add more detail as well!). In writing the documentation I also realized most of those cases weren't specifically tested so I added that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059532309
##########
cpp/src/parquet/api/reader.h:
##########
@@ -22,6 +22,7 @@
#include "parquet/column_scanner.h"
#include "parquet/exception.h"
#include "parquet/file_reader.h"
+#include "parquet/geospatial/statistics.h"
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969521161
##########
python/pyarrow/_parquet.pyx:
##########
@@ -312,6 +312,93 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeospatialStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return """{}
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007972712
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kWKBValueMin = 0,
Review Comment:
I changed these to `kValueMin/kValueMax` (I don't have any strong feelings on the naming here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2841197261
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 3a018c896a7fbfbfe9db2ee970d063c09905fc62.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/41405282742) has more details. It also includes information about 60 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2040427563
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::shared_ptr<const LogicalType>> ParseGeoArrowJSON(
+ std::string_view serialized_data);
+}
+
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ std::string_view serialized_data) {
+ // Handle a few hard-coded cases so that the tests can run/users can write the default
+ // LogicalType::Geometry() even if ARROW_JSON is not defined.
+ if (serialized_data.empty() || serialized_data == "{}") {
+ return LogicalType::Geometry();
+ } else if (
+ serialized_data ==
+ R"({"edges": "spherical", "crs": "OGC:CRS84", "crs_type": "authority_code"})") {
Review Comment:
I removed this...it was left over from when I wanted the tests to pass when ARROW_JSON was not ON. This is handled by the JSON parser now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016140395
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
Review Comment:
Can we use `Invalid` instead of `SerializationError` (which is not used commonly for parse errors)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016162989
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
Review Comment:
We should prevent the user from calling `GeoStatistics()` because the returned object would be crashy. There are similar patterns elsewhere in the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059538244
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ const std::string kSridPrefix{"srid:"};
+ const std::string kProjjsonPrefix{"projjson:"};
+
+ // Two reccomendataions are explicitly mentioned in the Parquet format for the
+ // LogicalType crs:
+ //
+ // - "srid:XXXX" as a way to encode an application-specific integer identifier
+ // - "projjson:some_field_name" as a way to avoid repeating PROJJSON strings
+ // unnecessarily (with a suggestion to place them in the file metadata)
+ //
+ // While we don't currently generate those values to reduce the complexity
+ // of the writer, we do interpret these values according to the suggestion in
+ // the format and pass on this information to GeoArrow.
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (::arrow::internal::StartsWith(crs, kSridPrefix)) {
+ return R"("crs": ")" + crs.substr(kSridPrefix.size()) + R"(", "crs_type": "srid")";
+ } else if (::arrow::internal::StartsWith(crs, kProjjsonPrefix)) {
+ std::string metadata_field = crs.substr(kProjjsonPrefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
Review Comment:
Good call! I reused the mechanism from a few lines below that checks for valid JSON (and escapes it as a JSON string if it's not)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059542503
##########
cpp/src/parquet/geospatial/statistics_test.cc:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial/statistics.h"
+#include "parquet/test_util.h"
+
+static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+namespace parquet::geospatial {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_TRUE(stats.geometry_types().has_value());
+ EXPECT_EQ(stats.geometry_types().value().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.dimension_empty(), ::testing::ElementsAre(true, true, true, true));
+ EXPECT_THAT(stats.dimension_valid(), ::testing::ElementsAre(true, true, true, true));
+ for (int i = 0; i < kMaxDimensions; i++) {
+ EXPECT_EQ(stats.lower_bound()[i], kInf);
+ EXPECT_EQ(stats.upper_bound()[i], -kInf);
+ }
+
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
Review Comment:
I added a dedicated test explicitly setting various components to be unequal!
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,174 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(std::shared_ptr<const LogicalType> type, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(
+ PrimitiveNode::Make("g", Repetition::REQUIRED, type, Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64);
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer =
+ ::arrow::internal::checked_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointXYSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+ std::memcpy(ptr, item.data(), item.size());
+ values[k].len = test::kWkbPointXYSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointXYSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ for (int k = 0; k < kNumRows; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+
+ ASSERT_OK(builder.Append(item));
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(),
+ /*leaf_field_nullable=*/true));
+ }
+
+ void TestWriteAndRead(std::shared_ptr<const LogicalType> type, bool write_arrow) {
+ ASSERT_NO_FATAL_FAILURE(WriteTestData(type, write_arrow));
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ ASSERT_TRUE(type->Equals(*metadata->schema()->Column(0)->logical_type()));
+
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ int64_t start_index = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geo_stats = column_chunk_metadata->geo_statistics();
+ ASSERT_NO_FATAL_FAILURE(CheckGeoStatistics(type, geo_stats, start_index,
+ row_group_metadata->num_rows()));
+ start_index += row_group_metadata->num_rows();
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ auto xy = test::GetWKBPointCoordinateXY(value);
+ EXPECT_TRUE(xy.has_value());
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_EQ(*xy, (std::pair<double, double>(expected_x, expected_y)));
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeoStatistics(std::shared_ptr<const LogicalType> type,
+ std::shared_ptr<geospatial::GeoStatistics> geom_stats,
+ int64_t start_index, int64_t num_rows) {
+ // We don't yet generate statistics for Geography
+ if (type->is_geography()) {
+ ASSERT_EQ(geom_stats, nullptr);
+ return;
+ }
+
+ ASSERT_NE(geom_stats, nullptr);
+ // We wrote exactly one geometry type (POINT, which has code 1)
+ EXPECT_THAT(*geom_stats->geometry_types(), ::testing::ElementsAre(1));
+
+ double expected_xmin = static_cast<double>(start_index);
+ double expected_xmax = expected_xmin + num_rows - 1;
+ double expected_ymin = expected_xmin + 1;
+ double expected_ymax = expected_xmax + 1;
+
+ EXPECT_EQ(geom_stats->lower_bound()[0], expected_xmin);
+ EXPECT_EQ(geom_stats->upper_bound()[0], expected_xmax);
+ EXPECT_EQ(geom_stats->lower_bound()[1], expected_ymin);
+ EXPECT_EQ(geom_stats->upper_bound()[1], expected_ymax);
+ EXPECT_THAT(geom_stats->dimension_valid(),
+ ::testing::ElementsAre(true, true, false, false));
+ }
+
+ protected:
+ std::shared_ptr<Buffer> file_buf;
+};
+
+TEST_F(TestGeometryLogicalType, TestWriteGeometry) {
+ TestWriteAndRead(GeometryLogicalType::Make("srid:1234"), /*write_arrow=*/false);
+}
+
+TEST_F(TestGeometryLogicalType, TestWriteArrowAndReadGeometry) {
+ TestWriteAndRead(GeometryLogicalType::Make("srid:1234"), /*write_arrow=*/true);
+}
+
+TEST_F(TestGeometryLogicalType, TestWriteGeography) {
+ TestWriteAndRead(GeographyLogicalType::Make("srid:1234"), /*write_arrow=*/false);
Review Comment:
Added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057938726
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
Hmm, why did we introduce `PARQUET_MINIMAL_DEPENDENCY` for? What is the purpose? It doesn't seem used or tested anywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2067514069
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
Review Comment:
One can in theory make use of an empty dimension to prune row groups (e.g., pushing down the predicate `st_hasz()` to skip row groups that have no Z value)...this is needed to separate the "not provided"/"invalid" case. I'm happy to follow-up with a PR to iterate on these names...it is not a trivial concept to parameterize!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2024527380
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status MergeGeometry(std::string_view bytes_wkb);
Review Comment:
@wgtmac What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008350260
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.ReadGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status =
+ bounder_.ReadGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
Review Comment:
Removed!
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.ReadGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
Review Comment:
Removed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008344996
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double get_xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double get_xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
Review Comment:
Good catch! A version of spec towards the end had wraparound for both X and Y (I am very glad that the final spec did not include this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2680363368
> Create a separate branch for writing example files with arbitrary CRSes
Is it possible to add new [examples](https://github.com/apache/arrow/tree/main/cpp/examples/parquet) instead of creating a new branch? It is much simpler to link rapidjson to these example executables. Test files are also reproducible in this approach.
> Keep support for reading into GeoArrow.
I'm fine with this given that the integration is trivial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975881392
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974528181
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
Review Comment:
I added the experimental label here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975804227
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
Review Comment:
I didn't add this one in (it's not part of `Statistics` either) but could. This only gets used in a handful of tests.
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetXMax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
+ /// case, these bounds represent the union of the intervals [ymax, Inf] and [-Inf,
+ /// ymin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetYMin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no Y values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetYMax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMax() const;
+
+ /// \brief Returns true if any Z values were encountered or false otherwise
+ bool HasZ() const;
+
+ /// \brief Returns true if any M values were encountered or false otherwise
+ bool HasM() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> GetGeometryTypes() const;
+
+ private:
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975801831
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
Review Comment:
I updated Decode() to reset first, so it really is Decode now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975805508
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
Review Comment:
I couldn't find this for `Statistics` so I didn't add it, but could (this only gets used in a few places).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2691254227
One separate thing that would be nice to do as a follow up is adding a fuzzing test for the new parsers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975906378
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes = 21 + (has_z ? 8 : 0) + (has_m ? 8 : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::POINT, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
+
+ return wkb;
+}
+
+static constexpr int kWkbPointSize = 21; // 1:endianness + 4:type + 8:x + 8:y
+
+inline void GenerateWKBPoint(uint8_t* ptr, double x, double y) {
+ double xyzm[] = {x, y, geometry::kInf, geometry::kInf};
+ std::string wkb = MakeWKBPoint(xyzm, false, false);
+ std::memcpy(ptr, wkb.data(), kWkbPointSize);
+}
+
+inline bool GetWKBPointCoordinate(const ByteArray& value, double* out_x, double* out_y) {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1971168184
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
Review Comment:
This one gives me `No member named 'AppendValue' in 'arrow::BinaryBuilder'`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955164146
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
Review Comment:
I saw `Visit` in an Arrow header so I changed it to that (but happy to use something else if it's more clear!)
I will circle back to documentation this evening (it's a great point that there isn't any 😬 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955190059
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
Review Comment:
This is a great point! I'll circle back to this one this evening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953939352
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(std::array<double, 4> coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ double min[4];
+ double max[4];
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return std::memcmp(lhs.min, rhs.min, sizeof(lhs.min)) == 0 &&
+ std::memcmp(lhs.max, rhs.max, sizeof(lhs.max)) == 0;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src, bool record_wkb_type = true) {
Review Comment:
from an API perspective is it intended to let callers change record_wkb_type? If not consider make ReadGeometry without this parameter then move this implementation to a private helper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953933714
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
Review Comment:
it might be worth passing array > 3 byte reference (or more generally most of them by reference). I guess without a benchmark it might be hard to tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955564394
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+};
+
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src) { return ReadGeometryInternal(src); }
+
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ const BoundingBox& Bounds() const { return box_; }
+
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+ ::arrow::Status ReadGeometryInternal(WKBBuffer* src, bool record_wkb_type = true) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
+ ParseGeometryType(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type_and_dimensions.first) {
+ case GeometryType::POINT:
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, 1, swap));
+ break;
+
+ case GeometryType::LINESTRING: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ break;
+ }
+ case GeometryType::POLYGON: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::MULTIPOINT:
+ case GeometryType::MULTILINESTRING:
+ case GeometryType::MULTIPOLYGON:
+ case GeometryType::GEOMETRYCOLLECTION: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(src, /*record_wkb_type*/ false));
+ }
+ break;
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions dimensions, uint32_t n_coords,
+ bool swap) {
+ switch (dimensions) {
+ case Dimensions::XY:
+ return src->ReadDoubles<BoundingBox::XY>(
+ n_coords, swap, [&](BoundingBox::XY coord) { box_.UpdateXY(coord); });
+ case Dimensions::XYZ:
+ return src->ReadDoubles<BoundingBox::XYZ>(
+ n_coords, swap, [&](BoundingBox::XYZ coord) { box_.UpdateXYZ(coord); });
+ case Dimensions::XYM:
+ return src->ReadDoubles<BoundingBox::XYM>(
+ n_coords, swap, [&](BoundingBox::XYM coord) { box_.UpdateXYM(coord); });
+ case Dimensions::XYZM:
+ return src->ReadDoubles<BoundingBox::XYZM>(
+ n_coords, swap, [&](BoundingBox::XYZM coord) { box_.UpdateXYZM(coord); });
+ default:
+ return ::arrow::Status::Invalid("Unknown dimensions");
+ }
+ }
+
+ static ::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(
+ uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::POINT);
+ auto max_geometry_type_value =
+ static_cast<uint32_t>(GeometryType::GEOMETRYCOLLECTION);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::XY);
Review Comment:
it might be cleaner to have a specific Min/max member of MIN/MAX (I believe you can have two symbols pointing the same value).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953921774
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
Review Comment:
the data_ and size_updates seem to be sprinkled around in a lot of different places I wonder it it would pay to make a generic method like `template<typename T> T UnsafeConsume() {
T t = SafeLoadAs<T>(data_, sizeof(T))
data_ += sizeof(T);
size_ -= sizeof(T);
}
template<typename T> Result<T> Consume() {
if (sizeof(T) > size_) {
... return error
}
return UnsafeConsume<T>();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953911654
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
Review Comment:
please try to document classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953926664
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
Review Comment:
or maybe not nest this in a struct and just have the static methods here as top level functions? then GeometryType could be the enum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2050086670
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool writer_calculated_xy_bounds{};
+ double xmin{};
Review Comment:
Why not initialized to kNaN to minimize the chance of misuse? IMO, we should avoid emitting NaN to thrift but it is fine to use NaN at runtime to represent invalid data.
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool writer_calculated_xy_bounds{};
+ double xmin{};
+ double xmax{};
+ double ymin{};
+ double ymax{};
+
+ bool writer_calculated_z_bounds{};
Review Comment:
Can we simply rename it to `is_z_valid`?
##########
cpp/src/parquet/metadata.cc:
##########
@@ -299,13 +312,25 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
return false;
}
if (possible_stats_ == nullptr) {
+ // Because we are modifying possible_stats_ in a const method
+ const std::lock_guard<std::mutex> guard(stats_mutex_);
possible_stats_ = MakeColumnStats(*column_metadata_, descr_);
}
Review Comment:
```suggestion
{
// Because we are modifying possible_stats_ in a const method
const std::lock_guard<std::mutex> guard(stats_mutex_);
if (possible_stats_ == nullptr) {
possible_stats_ = MakeColumnStats(*column_metadata_, descr_);
}
}
```
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,393 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = !bound_empty(2) && bound_valid(2);
+ bool write_m = !bound_empty(3) && bound_valid(3);
Review Comment:
```suggestion
bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
```
Should we drop bbox entirely if x or y is missing as they are `required`?
##########
cpp/src/parquet/thrift_internal.h:
##########
@@ -232,6 +233,81 @@ static inline AadMetadata FromThrift(format::AesGcmCtrV1 aesGcmCtrV1) {
aesGcmCtrV1.supply_aad_prefix};
}
+static inline geospatial::EncodedGeoStatistics FromThrift(
+ const format::GeospatialStatistics& geo_stats) {
+ geospatial::EncodedGeoStatistics out;
+
+ out.geospatial_types = geo_stats.geospatial_types;
+
+ if (geo_stats.__isset.bbox) {
+ out.xmin = geo_stats.bbox.xmin;
+ out.xmax = geo_stats.bbox.xmax;
+ out.ymin = geo_stats.bbox.ymin;
+ out.ymax = geo_stats.bbox.ymax;
+ out.writer_calculated_xy_bounds = true;
+
+ if (geo_stats.bbox.__isset.zmin && geo_stats.bbox.__isset.zmax) {
+ out.zmin = geo_stats.bbox.zmin;
+ out.zmax = geo_stats.bbox.zmax;
+ out.writer_calculated_z_bounds = true;
Review Comment:
`writer_` prefix looks odd from the perspective of this function (which is on the reader side).
##########
cpp/src/generated/parquet_types.h:
##########
@@ -4,3888 +4,3888 @@
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
* @generated
*/
-#ifndef parquet_TYPES_H
-#define parquet_TYPES_H
+ #ifndef parquet_TYPES_H
Review Comment:
Why do we still see this file after rebased?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057888828
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
Do we need to modify the CMake config to enable `ARROW_JSON` when `ARROW_PARQUET` is ON? cc @kou @raulcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059540161
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax))) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ geospatial::BoundingBox box;
+
+ if (encoded.xy_bounds_present) {
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+ } else {
+ box.min[0] = kNaN;
+ box.max[0] = kNaN;
+ box.min[1] = kNaN;
+ box.max[1] = kNaN;
+ }
+
+ if (encoded.z_bounds_present) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ } else {
+ box.min[2] = kNaN;
+ box.max[2] = kNaN;
+ }
+
+ if (encoded.m_bounds_present) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ } else {
+ box.min[3] = kNaN;
+ box.max[3] = kNaN;
+ }
+
+ bounder_.MergeBox(box);
+ bounder_.MergeGeometryTypes(encoded.geospatial_types);
+ has_geometry_types_ = has_geometry_types_ && encoded.geospatial_types_present();
+ }
+
+ bool is_wraparound_x() const {
+ return is_wraparound(lower_bound()[0], upper_bound()[0]);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ bool bounds_empty() const {
+ for (int i = 0; i < kMaxDimensions; i++) {
+ if (!bound_empty(i)) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ bool bound_empty(int i) const {
+ return std::isinf(bounder_.Bounds().min[i] - bounder_.Bounds().max[i]);
+ }
+
+ bool bound_valid(int i) const {
+ return !std::isnan(bounder_.Bounds().min[i]) && !std::isnan(bounder_.Bounds().max[i]);
+ }
+
+ const std::array<double, kMaxDimensions>& lower_bound() const {
+ return bounder_.Bounds().min;
+ }
+
+ const std::array<double, kMaxDimensions>& upper_bound() const {
+ return bounder_.Bounds().max;
+ }
+
+ std::optional<std::vector<int32_t>> geometry_types() const {
+ if (has_geometry_types_) {
+ return bounder_.GeometryTypes();
+ } else {
+ return std::nullopt;
+ }
+ }
+
+ private:
+ geospatial::WKBGeometryBounder bounder_;
+ bool is_valid_ = true;
+ bool has_geometry_types_ = true;
+
+ template <typename ArrayType>
+ void UpdateArrayImpl(const ::arrow::Array& values) {
+ const auto& binary_array = static_cast<const ArrayType&>(values);
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
+ if (!binary_array.IsNull(i)) {
+ try {
+ bounder_.MergeGeometry(binary_array.GetView(i));
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+ }
+
+ static bool is_wraparound(double dmin, double dmax) {
Review Comment:
I added a note here. It was added to the spec to provide an opportunity for something like Fiji to have an X bound like [179.7, -179.3] instead of the much less useful [-180, 180] (because Fiji happens to be on top of the antimeridian).
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax))) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ geospatial::BoundingBox box;
+
+ if (encoded.xy_bounds_present) {
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+ } else {
+ box.min[0] = kNaN;
+ box.max[0] = kNaN;
+ box.min[1] = kNaN;
+ box.max[1] = kNaN;
+ }
+
+ if (encoded.z_bounds_present) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ } else {
+ box.min[2] = kNaN;
+ box.max[2] = kNaN;
+ }
+
+ if (encoded.m_bounds_present) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ } else {
+ box.min[3] = kNaN;
+ box.max[3] = kNaN;
+ }
+
+ bounder_.MergeBox(box);
+ bounder_.MergeGeometryTypes(encoded.geospatial_types);
+ has_geometry_types_ = has_geometry_types_ && encoded.geospatial_types_present();
+ }
+
+ bool is_wraparound_x() const {
+ return is_wraparound(lower_bound()[0], upper_bound()[0]);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ bool bounds_empty() const {
+ for (int i = 0; i < kMaxDimensions; i++) {
+ if (!bound_empty(i)) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ bool bound_empty(int i) const {
+ return std::isinf(bounder_.Bounds().min[i] - bounder_.Bounds().max[i]);
+ }
+
+ bool bound_valid(int i) const {
+ return !std::isnan(bounder_.Bounds().min[i]) && !std::isnan(bounder_.Bounds().max[i]);
+ }
+
+ const std::array<double, kMaxDimensions>& lower_bound() const {
+ return bounder_.Bounds().min;
+ }
+
+ const std::array<double, kMaxDimensions>& upper_bound() const {
+ return bounder_.Bounds().max;
+ }
+
+ std::optional<std::vector<int32_t>> geometry_types() const {
+ if (has_geometry_types_) {
+ return bounder_.GeometryTypes();
+ } else {
+ return std::nullopt;
+ }
+ }
+
+ private:
+ geospatial::WKBGeometryBounder bounder_;
+ bool is_valid_ = true;
+ bool has_geometry_types_ = true;
+
+ template <typename ArrayType>
+ void UpdateArrayImpl(const ::arrow::Array& values) {
+ const auto& binary_array = static_cast<const ArrayType&>(values);
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
+ if (!binary_array.IsNull(i)) {
+ try {
+ bounder_.MergeGeometry(binary_array.GetView(i));
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+ }
+
+ static bool is_wraparound(double dmin, double dmax) {
+ return !std::isinf(dmin - dmax) && dmin > dmax;
+ }
+};
+
+GeoStatistics::GeoStatistics() : impl_(std::make_unique<GeoStatisticsImpl>()) {}
+
+GeoStatistics::GeoStatistics(const EncodedGeoStatistics& encoded) : GeoStatistics() {
+ Decode(encoded);
+}
+
+GeoStatistics::~GeoStatistics() = default;
+
+bool GeoStatistics::Equals(const GeoStatistics& other) const {
+ return impl_->Equals(*other.impl_);
+}
+
+void GeoStatistics::Merge(const GeoStatistics& other) { impl_->Merge(*other.impl_); }
+
+void GeoStatistics::Update(const ByteArray* values, int64_t num_values) {
+ impl_->Update(values, num_values);
+}
+
+void GeoStatistics::UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ impl_->UpdateSpaced(values, valid_bits, valid_bits_offset, num_spaced_values,
+ num_values);
+}
+
+void GeoStatistics::Update(const ::arrow::Array& values) { impl_->Update(values); }
+
+void GeoStatistics::Reset() { impl_->Reset(); }
+
+bool GeoStatistics::is_valid() const { return impl_->is_valid(); }
+
+std::optional<EncodedGeoStatistics> GeoStatistics::Encode() const {
+ if (is_valid()) {
+ return impl_->Encode();
+ } else {
+ return std::nullopt;
+ }
+}
+
+void GeoStatistics::Decode(const EncodedGeoStatistics& encoded) {
+ impl_->Reset();
+ impl_->Update(encoded);
+}
+
+std::array<double, 4> GeoStatistics::lower_bound() const { return impl_->lower_bound(); }
+
+std::array<double, 4> GeoStatistics::upper_bound() const { return impl_->upper_bound(); }
+
+std::array<bool, 4> GeoStatistics::dimension_empty() const {
+ return {impl_->bound_empty(0), impl_->bound_empty(1), impl_->bound_empty(2),
+ impl_->bound_empty(3)};
+}
+
+std::array<bool, 4> GeoStatistics::dimension_valid() const {
+ return {impl_->bound_valid(0), impl_->bound_valid(1), impl_->bound_valid(2),
+ impl_->bound_valid(3)};
+}
+
+std::optional<std::vector<int32_t>> GeoStatistics::geometry_types() const {
+ return impl_->geometry_types();
+}
+
+std::string GeoStatistics::ToString() const {
+ if (!is_valid()) {
+ return "GeoStatistics <invalid>\n";
+ }
+
+ std::stringstream ss;
+ ss << "GeoStatistics " << std::endl;
Review Comment:
I updated this to be a one-liner!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059534202
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool writer_calculated_geospatial_types() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief All minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics imported from Thrift.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Returns true if the dimension is valid and any non-NaN values were
+ /// encountered in the given dimension in XYZM order
+ std::array<bool, kMaxDimensions> dimension_empty() const;
Review Comment:
I went through and added more detail and some examples in all of these docstrings (and fixed this one, which was the exact opposite, as you noted).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059539139
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1343,6 +1390,96 @@ TEST_F(TestConvertArrowSchema, ParquetFlatPrimitivesAsDictionaries) {
ASSERT_NO_FATAL_FAILURE(CheckFlatSchema(parquet_fields));
}
+TEST_F(TestConvertArrowSchema, ParquetGeoArrowCrsLonLat) {
+ // All the Arrow Schemas below should convert to the type defaults for GEOMETRY
+ // and GEOGRAPHY when GeoArrow extension types are registered and the appropriate
+ // writer option is set.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ // There are several ways that longitude/latitude could be specified when coming from
+ // GeoArrow, which allows null, missing, arbitrary strings (e.g., Authority:Code), and
+ // PROJJSON.
+ std::vector<std::string> geoarrow_lonlat = {
+ "null", R"("OGC:CRS84")", R"("EPSG:4326")",
+ // Purely the parts of the PROJJSON that we inspect to check the lon/lat case
+ R"({"id": {"authority": "OGC", "code": "CRS84"}})",
+ R"({"id": {"authority": "EPSG", "code": 4326}})"};
Review Comment:
I added support for this as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955155167
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
Review Comment:
It's because ISO WKB defined geometry types such that `/ 1000` and `% 1000` can be used to separate the geometry type and dimensions component. I moved the `/ 1000` and `% 1000` next to eachother and added a comment because I wasn't sure what exactly to name the constant but I'm open to suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2038875059
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1230,11 +1242,19 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
current_dict_encoder_ =
dynamic_cast<DictEncoder<ParquetType>*>(current_encoder_.get());
+ // Geometry/Geography are the first non-nested logical types to have a
+ // SortOrder::UNKNOWN. Currently, the presence of statistics is tied to
+ // having a known sort order and so null counts will be missing.
if (properties->statistics_enabled(descr_->path()) &&
(SortOrder::UNKNOWN != descr_->sort_order())) {
page_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_);
chunk_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_);
+ } else if (properties->statistics_enabled(descr_->path()) &&
Review Comment:
What above the below to avoid checking `statistics_enabled` twice:
```
if (properties->statistics_enabled(descr_->path())) {
if (SortOrder::UNKNOWN != descr_->sort_order()) {
page_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_);
chunk_statistics_ = MakeStatistics<ParquetType>(descr_, allocator_);
}
if (descr_->logical_type() != nullptr &&
descr_->logical_type()->is_geometry()) {
chunk_geospatial_statistics_ = std::make_shared<GeoStatistics>();
}
}
```
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_empty() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
Review Comment:
```suggestion
return !(!geospatial_types.empty() || has_x() || has_y() || has_z() || has_m());
```
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +321,14 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geo_stats_set() const {
+ if (possible_geo_stats_ == nullptr &&
+ column_metadata_->__isset.geospatial_statistics) {
+ possible_geo_stats_ = MakeColumnGeometryStats(*column_metadata_, descr_);
Review Comment:
Is it worth adding a mutex for both `is_stats_set` and `is_geo_stats_set`?
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::shared_ptr<const LogicalType>> ParseGeoArrowJSON(
+ std::string_view serialized_data);
+}
+
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ std::string_view serialized_data) {
+ // Handle a few hard-coded cases so that the tests can run/users can write the default
+ // LogicalType::Geometry() even if ARROW_JSON is not defined.
+ if (serialized_data.empty() || serialized_data == "{}") {
+ return LogicalType::Geometry();
+ } else if (
+ serialized_data ==
+ R"({"edges": "spherical", "crs": "OGC:CRS84", "crs_type": "authority_code"})") {
Review Comment:
Does it require some sort of normalization just in case there are spaces or keys are in a different order?
##########
cpp/src/parquet/geospatial/statistics_test.cc:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial/statistics.h"
+#include "parquet/test_util.h"
+
+static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+namespace parquet::geometry {
Review Comment:
Why some files (e.g. geospatial/statistics.h/.cc) are in the `namespace parquet` while others are in the `namespace parquet::geometry`? Should we be consistent? And should it be `namespace parquet::geospatial` instead if we want a nested one?
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -373,6 +385,8 @@ add_parquet_test(internals-test
statistics_test.cc
encoding_test.cc
metadata_test.cc
+ geospatial/statistics_test.cc
Review Comment:
Sort alphabetically?
##########
cpp/src/parquet/geospatial/util_internal.h:
##########
@@ -0,0 +1,216 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <array>
+#include <cmath>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging_internal.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
Review Comment:
It has a duplicate definition in the `statistics.h`
##########
cpp/src/parquet/parquet.thrift:
##########
@@ -281,7 +281,7 @@ struct Statistics {
*/
1: optional binary max;
2: optional binary min;
- /**
+ /**
Review Comment:
We can rebase to get rid of changes on these files now.
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,369 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geometry::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geometry::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+ out.geospatial_types = bounder_.GeometryTypes();
+
+ if (!bound_empty(0)) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ }
+
+ if (!bound_empty(1)) {
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ }
+
+ if (!bound_empty(2)) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ }
+
+ if (!bound_empty(3)) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax) ||
+ is_wraparound(encoded.ymin, encoded.ymax))) {
Review Comment:
```suggestion
(is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax))) {
```
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,369 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geometry::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geometry::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+ out.geospatial_types = bounder_.GeometryTypes();
+
+ if (!bound_empty(0)) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ }
+
+ if (!bound_empty(1)) {
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ }
+
+ if (!bound_empty(2)) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ }
+
+ if (!bound_empty(3)) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
+ if (!bounds_empty() &&
+ (is_wraparound_x() || is_wraparound(encoded.xmin, encoded.xmax) ||
+ is_wraparound(encoded.ymin, encoded.ymax))) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ geometry::BoundingBox box;
+
+ if (encoded.has_x()) {
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ }
+
+ if (encoded.has_y()) {
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+ }
+
+ if (encoded.has_z()) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ }
+
+ if (encoded.has_m()) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ }
+
+ bounder_.MergeBox(box);
+ bounder_.MergeGeometryTypes(encoded.geospatial_types);
+ }
+
+ bool is_wraparound_x() const {
+ return is_wraparound(lower_bound()[0], upper_bound()[0]);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ bool bounds_empty() const {
+ for (int i = 0; i < 4; i++) {
Review Comment:
Should we declare `constexpr int kNumDimensions = 4` and use it here and below for `std::array<double, 4>`?
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -788,6 +788,9 @@ class ColumnWriterImpl {
// Plain-encoded statistics of the whole chunk
virtual StatisticsPair GetChunkStatistics() = 0;
+ // Plain-encoded geometry statistics of the whole chunk
Review Comment:
```suggestion
// Geospatial statistics of the whole chunk
```
This has nothing to do with plain encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007953070
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -1088,6 +1114,10 @@ Status ToParquetSchema(const ::arrow::Schema* arrow_schema,
const WriterProperties& properties,
const ArrowWriterProperties& arrow_properties,
std::shared_ptr<SchemaDescriptor>* out) {
+ // TODO(paleolimbot): I'm wondering if this geo_crs_context is reused when testing
+ // on MINGW, where we get some failures indicating non-empty metadata where it was
+ // expected
+ arrow_properties.geo_crs_context()->Clear();
Review Comment:
I had to add this because a number of tests reused the `arrow_properties` (sometimes) so I had to clear this before each walk of the schema. I agree this is weird (which is one of the reasons I removed the `GeoCrsContext`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970043706
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
Review Comment:
Yes, the geography bounding is not implemented in this PR. I added some checks (and tested the checks) to make sure we don't accidentally ingest encoded statistics (which might contain a wraparound) and try to merge them using logic that isn't implemented yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975827701
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
Review Comment:
I added a check!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975790699
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status = bounder_.ReadGeometry(item.ptr, item.len);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(item.ptr, item.len));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException(
+ "Unsupported Array type in GeospatialStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeospatialStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geometry::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geometry::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeospatialStatistics out;
+ out.geospatial_types = bounder_.GeometryTypes();
+
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.zmin = mins[2];
Review Comment:
Good call! I now check for the existence of Z and M here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975896373
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ double x = 0;
+ double y = 0;
+ EXPECT_TRUE(test::GetWKBPointCoordinate(value, &x, &y));
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_DOUBLE_EQ(expected_x, x);
+ EXPECT_DOUBLE_EQ(expected_y, y);
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeospatialStatistics(std::shared_ptr<GeospatialStatistics> geom_stats) {
+ ASSERT_TRUE(geom_stats != nullptr);
+ std::vector<int32_t> geospatial_types = geom_stats->GetGeometryTypes();
+ EXPECT_EQ(1, geospatial_types.size());
+ EXPECT_EQ(1, geospatial_types[0]);
Review Comment:
I added a comment here (1 is the WKB integer type code for POINT (without Z or M))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975901815
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
Review Comment:
I removed this! An earlier version of the specification wrote page statistics also (this provision was removed and now statistics are only written at the column chunk level, like size statistics).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969522997
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
+
+ private:
+ GeographyLogicalType() = default;
Review Comment:
All of the other logical types use this...if it's no longer useful I wonder if it would be appropriate to change all of that in a separate change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016153795
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1658,209 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+namespace {
+void WriteCrsKeyAndValue(const std::string& crs, std::ostream& json) {
+ // There is no restriction on the crs value here, and it may contain quotes
+ // or backslashes that would result in invalid JSON if unescaped.
+ json << R"(, "crs": ")";
+ for (char c : crs) {
+ switch (c) {
+ case '"':
+ json << R"(\")";
+ break;
+ case '\\':
+ json << R"(\\)";
+ break;
+ default:
Review Comment:
Characters below 32 should also be encoded as unicode escapes (`\uXXXX`), I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017016290
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
> Then it's a bug that would deserve fixing :) Do you want to look into that?
Of course, let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016179107
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1104,9 +1107,18 @@ int64_t ColumnWriterImpl::Close() {
if (rows_written_ > 0 && chunk_statistics.is_set()) {
metadata_->SetStatistics(chunk_statistics);
}
+
if (rows_written_ > 0 && chunk_size_statistics.is_set()) {
metadata_->SetSizeStatistics(chunk_size_statistics);
}
+
+ if (descr_->logical_type() != nullptr && descr_->logical_type()->is_geometry()) {
Review Comment:
I'm curious, only geometry columns can have geostats, not geography columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017472016
##########
cpp/src/parquet/api/reader.h:
##########
@@ -22,6 +22,7 @@
#include "parquet/column_scanner.h"
#include "parquet/exception.h"
#include "parquet/file_reader.h"
+#include "parquet/geospatial_statistics.h"
Review Comment:
Yes, I believe we want/need this because a C++ client can now access the `GeoStatistics` from the column metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017866549
##########
cpp/src/parquet/test_util.h:
##########
@@ -39,6 +39,7 @@
#include "parquet/column_reader.h"
#include "parquet/column_writer.h"
#include "parquet/encoding.h"
+#include "parquet/geospatial_util_internal.h"
Review Comment:
I removed this!
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017865379
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ return list(self.statistics.geospatial_types)
+
+ @property
+ def xmin(self):
+ return self.statistics.xmin
Review Comment:
I updated this! (`has_x()` and `has_y()` were added after this had been written)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016168034
##########
cpp/src/parquet/test_util.cc:
##########
@@ -194,5 +194,84 @@ void prefixed_random_byte_array(int n, uint32_t seed, uint8_t* buf, int len, FLB
}
}
+namespace {
+
+uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z, bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+} // namespace
+
+std::string MakeWKBPoint(const std::vector<double>& xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes =
+ kWkbPointXYSize + (has_z ? sizeof(double) : 0) + (has_m ? sizeof(double) : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::kPoint, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
Review Comment:
```suggestion
ptr += 8;
}
DCHECK_EQ(ptr - wkb.data(), wkb.length());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955195459
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
Review Comment:
I'm happy to pull out the implementations into a .cc file although I wonder if this is slightly easier to drop in to the 3 or 4 other C++ Parquet implementations if kept together. I would also wonder if the compiler benefits from seeing the implementations (but I'm no expert here!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2067511281
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
+ ///
+ /// When calculating statistics, zero or more of these values may be true because
+ /// this implementation calculates bounds for all dimensions; however, it may be
+ /// true that zero coordinates were encountered in a given dimension. For example,
+ /// dimension_empty() will return four true values if Update() was not called
+ /// or if Update() was called with only null values. If Update() was provided
+ /// one or more geometries with X and Y dimensions but not Z or M dimensions,
+ /// dimension_empty() will return true, true, false, false.
+ ///
+ /// For statistics read from a Parquet file, dimension_empty() will always contain
+ /// false values because there is no mechanism to communicate an empty interval
+ /// in the Thrift metadata.
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Dimension validity (i.e. presence) in XYZM order
+ ///
+ /// When calculating statistics, this will always be true because this implementation
+ /// calculates statistics for all dimensions. When reading a Parquet file, one or more
+ /// of these values may be false because the file may not have provided bounds for all
+ /// dimensions.
+ ///
+ /// In other words, it is safe to use dimension_empty(), lower_bound(), and/or
Review Comment:
I reworded this comment...there are documented canonical values for those functions for the invalid per-dimension case!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2066448430
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
Review Comment:
If it's called `dimension_empty`, I would expect it to return `false` if there are indeed usable statistics. So you probably want to change the method name?
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
+ ///
+ /// When calculating statistics, zero or more of these values may be true because
+ /// this implementation calculates bounds for all dimensions; however, it may be
+ /// true that zero coordinates were encountered in a given dimension. For example,
+ /// dimension_empty() will return four true values if Update() was not called
+ /// or if Update() was called with only null values. If Update() was provided
+ /// one or more geometries with X and Y dimensions but not Z or M dimensions,
+ /// dimension_empty() will return true, true, false, false.
+ ///
+ /// For statistics read from a Parquet file, dimension_empty() will always contain
+ /// false values because there is no mechanism to communicate an empty interval
+ /// in the Thrift metadata.
+ std::array<bool, kMaxDimensions> dimension_empty() const;
+
+ /// \brief Dimension validity (i.e. presence) in XYZM order
+ ///
+ /// When calculating statistics, this will always be true because this implementation
+ /// calculates statistics for all dimensions. When reading a Parquet file, one or more
+ /// of these values may be false because the file may not have provided bounds for all
+ /// dimensions.
+ ///
+ /// In other words, it is safe to use dimension_empty(), lower_bound(), and/or
Review Comment:
TBH, it seems a bit weird that one can't call `dimension_empty` if `dimension_valid` is false. Perhaps we can make things simpler for the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059179655
##########
cpp/src/parquet/geospatial/util_json_internal.cc:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_json_internal.h"
+
+#include <string>
+
+#include "arrow/extension_type.h"
+#include "arrow/json/rapidjson_defs.h" // IWYU pragma: keep
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include <rapidjson/document.h>
+#include <rapidjson/writer.h>
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+namespace {
+::arrow::Result<std::string> GeospatialGeoArrowCrsToParquetCrs(
+ const ::arrow::rapidjson::Document& document) {
+ namespace rj = ::arrow::rapidjson;
+
+ if (!document.HasMember("crs") || document["crs"].IsNull()) {
+ // Parquet GEOMETRY/GEOGRAPHY do not have a concept of a null/missing
+ // CRS, but an omitted one is more likely to have meant "lon/lat" than
+ // a truly unspecified one (i.e., Engineering CRS with arbitrary XY units)
+ return "";
+ }
+
+ const auto& json_crs = document["crs"];
+ if (json_crs.IsString() && (json_crs == "EPSG:4326" || json_crs == "OGC:CRS84")) {
+ // crs can be left empty because these cases both correspond to
+ // longitude/latitude in WGS84 according to the Parquet specification
+ return "";
+ } else if (json_crs.IsObject()) {
+ // Attempt to detect common PROJJSON representations of longitude/latitude and return
+ // an empty crs to maximize compatibility with readers that do not implement CRS
+ // support. PROJJSON stores this in the "id" member like:
+ // {..., "id": {"authority": "...", "code": "..."}}
+ if (json_crs.HasMember("id")) {
+ const auto& identifier = json_crs["id"];
+ if (identifier.HasMember("authority") && identifier.HasMember("code")) {
+ if (identifier["authority"] == "OGC" && identifier["code"] == "CRS84") {
+ return "";
+ } else if (identifier["authority"] == "EPSG" && identifier["code"] == 4326) {
+ return "";
+ }
+ }
+ }
+ }
+
+ // If we could not detect a longitude/latitude CRS, just write the string to the
+ // LogicalType crs (being sure to unescape a JSON string into a regular string)
+ if (json_crs.IsString()) {
+ return json_crs.GetString();
+ } else {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ json_crs.Accept(writer);
+ return buffer.GetString();
+ }
+}
+
+::arrow::Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ const std::string kSridPrefix{"srid:"};
+ const std::string kProjjsonPrefix{"projjson:"};
+
+ // Two reccomendataions are explicitly mentioned in the Parquet format for the
+ // LogicalType crs:
+ //
+ // - "srid:XXXX" as a way to encode an application-specific integer identifier
+ // - "projjson:some_field_name" as a way to avoid repeating PROJJSON strings
+ // unnecessarily (with a suggestion to place them in the file metadata)
+ //
+ // While we don't currently generate those values to reduce the complexity
+ // of the writer, we do interpret these values according to the suggestion in
+ // the format and pass on this information to GeoArrow.
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (::arrow::internal::StartsWith(crs, kSridPrefix)) {
+ return R"("crs": ")" + crs.substr(kSridPrefix.size()) + R"(", "crs_type": "srid")";
+ } else if (::arrow::internal::StartsWith(crs, kProjjsonPrefix)) {
+ std::string metadata_field = crs.substr(kProjjsonPrefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ }
+ }
+
+ // Pass on the string directly to GeoArrow. If the string is already valid JSON,
+ // insert it directly into GeoArrow's "crs" field. Otherwise, escape it and pass it as a
+ // string value.
+ namespace rj = ::arrow::rapidjson;
+ rj::Document document;
+ if (document.Parse(crs.data(), crs.length()).HasParseError()) {
+ rj::StringBuffer buffer;
+ rj::Writer<rj::StringBuffer> writer(buffer);
+ writer.String(crs);
+ return R"("crs": )" + std::string(buffer.GetString());
+ } else {
+ return R"("crs": )" + crs;
+ }
+}
+
+} // namespace
+
+::arrow::Result<std::shared_ptr<const LogicalType>> LogicalTypeFromGeoArrowMetadata(
+ std::string_view serialized_data) {
+ // Parquet has no way to interpret a null or missing CRS, so we choose the most likely
+ // intent here (that the user meant to use the default Parquet CRS)
+ if (serialized_data.empty() || serialized_data == "{}") {
+ return LogicalType::Geometry();
+ }
+
+ namespace rj = ::arrow::rapidjson;
+ rj::Document document;
+ if (document.Parse(serialized_data.data(), serialized_data.length()).HasParseError()) {
+ return ::arrow::Status::Invalid("Invalid serialized JSON data: ", serialized_data);
+ }
+
+ ARROW_ASSIGN_OR_RAISE(std::string crs, GeospatialGeoArrowCrsToParquetCrs(document));
+
+ if (document.HasMember("edges") && document["edges"] == "planar") {
+ return LogicalType::Geometry(crs);
+ } else if (document.HasMember("edges") && document["edges"] == "spherical") {
+ return LogicalType::Geography(crs,
+ LogicalType::EdgeInterpolationAlgorithm::SPHERICAL);
+ } else if (document.HasMember("edges")) {
+ return ::arrow::Status::Invalid("Unsupported GeoArrow edge type: ", serialized_data);
+ }
+
+ return LogicalType::Geometry(crs);
+}
+
+::arrow::Result<std::shared_ptr<::arrow::DataType>> GeoArrowTypeFromLogicalType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ // Check if we have a registered GeoArrow type to read into
+ std::shared_ptr<::arrow::ExtensionType> maybe_geoarrow_wkb =
+ ::arrow::GetExtensionType("geoarrow.wkb");
Review Comment:
It does! (Parquet did too when the negotiations about adding it to the spec started, just not when they ended 🙂 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059536265
##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -0,0 +1,238 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/util_internal.h"
+
+#include <sstream>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geospatial {
+
+std::string BoundingBox::ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+}
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ uint8_t ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ uint32_t ReadUInt32(bool swap) {
+ auto value = ReadChecked<uint32_t>();
+ if (swap) {
+ return ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ void ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955187787
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(std::array<double, 4> coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ double min[4];
+ double max[4];
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return std::memcmp(lhs.min, rhs.min, sizeof(lhs.min)) == 0 &&
+ std::memcmp(lhs.max, rhs.max, sizeof(lhs.max)) == 0;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src, bool record_wkb_type = true) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type, GeometryType::FromWKB(wkb_geometry_type));
+ ARROW_ASSIGN_OR_RAISE(auto dimensions, Dimensions::FromWKB(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type) {
+ case GeometryType::geometry_type::POINT:
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, 1, swap));
+ break;
+
+ case GeometryType::geometry_type::LINESTRING: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, n_coords, swap));
+ break;
+ }
+ case GeometryType::geometry_type::POLYGON: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::geometry_type::MULTIPOINT:
+ case GeometryType::geometry_type::MULTILINESTRING:
+ case GeometryType::geometry_type::MULTIPOLYGON:
+ case GeometryType::geometry_type::GEOMETRYCOLLECTION: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometry(src, /*record_wkb_type*/ false));
+ }
+ break;
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ const BoundingBox& Bounds() const { return box_; }
+
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions::dimensions dimensions,
+ uint32_t n_coords, bool swap) {
+ using XY = std::array<double, 2>;
Review Comment:
I moved these into `BoundingBox::XY[Z[M]]`!
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(std::array<double, 4> coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ double min[4];
+ double max[4];
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return std::memcmp(lhs.min, rhs.min, sizeof(lhs.min)) == 0 &&
+ std::memcmp(lhs.max, rhs.max, sizeof(lhs.max)) == 0;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src, bool record_wkb_type = true) {
Review Comment:
I moved this to be internal!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955566061
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ if (!is_valid_ || !other.is_valid_) {
+ is_valid_ = false;
+ return;
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ buf.Init(item.ptr, item.len);
+ ::arrow::Status status = bounder_.ReadGeometry(&buf);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ buf.Init(item.ptr, item.len);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(&buf));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ const auto& binary_array = static_cast<const ::arrow::BinaryArray&>(values);
Review Comment:
it would be nice to check the type before casting. What about LargeBinary/StringView types? (I thought stringview had a binary equivelant)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955559494
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
Review Comment:
Not an expert either, but I think as long as long as the .h/.cc file are well isolated I hope other implementations won't feel the need to reinvent the wheel (I guess using Status might be the primary detractor).
In terms of inlining, the rule of thumb is generally less then 10 lines of code in any particular function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017854863
##########
cpp/src/parquet/api/reader.h:
##########
@@ -22,6 +22,7 @@
#include "parquet/column_scanner.h"
#include "parquet/exception.h"
#include "parquet/file_reader.h"
+#include "parquet/geospatial_statistics.h"
Review Comment:
(This is also the mechanism by which the `Statistics` are made available to wrappers, including pyarrow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017867021
##########
cpp/src/parquet/geospatial_util_json_internal.h:
##########
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
Review Comment:
I added to the comment here (it's the `ARROW:extension:metadata` for GeoArrow, which is serialized as JSON).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2749852558
Thank you for the review!
> To confirm, the segfault I mentioned is no longer present with the latest change to remove GeoCrsContext handling.
Yes, I think previously there was a `throw ParquetException(...)` that should have been a `return Status(...)`. I believe that was due to the bug whose fix you merged in GeoPandas (it was generating a string containing JSON and not a JSON object, and the previous iteration only supported JSON objects and threw the error when it saw a string). The bug now no longer affects this code, because it writes the stringified JSON to the LogicalType crs (the same result as serializing the RapidJSON object into the LogicalType crs).
> for the Parquet code to "see" a geometry column in the Arrow data it is writing, it needs to be an actual registered extension type, and having the extension metadata in the field metadata is not sufficient?
Yes. I believe we have access to the `Field` there and this is theoretically possible. I was going for a degree of "opt-in" here since this is all new (an earlier version of this PR exposed an option `write_geospatial_types`, which is another option). I really fine with anything here but I do thinking having some version of opt-in would be good for the first release that this lands in.
I think a better long-term solution would be a canonical extension type (i.e., always registered), and while I'd prefer to merge this as experimental and have that discussion over the next few months, I'm happy to implement any consensus we end up on here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2014481983
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
Review Comment:
> where a writer generated NaNs
If that happens, is the bbox still useful for PPD? Or we can only drop the stats?
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -259,6 +262,12 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # TODO(paleolimbot): Make sure this is OK or remove if not!
+ if(ARROW_JSON)
Review Comment:
I agree that making the change as minimal as possible is also important. I'm fine with this. @pitrou @emkornfield WDYT?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,363 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
Review Comment:
```suggestion
#include "parquet/geospatial_statistics.h"
```
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
Review Comment:
```suggestion
WKBBuffer() : data_(nullptr), size_(0) {}
```
We only need to use NULLPTR in the header files.
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
Review Comment:
```suggestion
"Can't read coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
```
##########
cpp/src/parquet/column_writer_test.cc:
##########
@@ -1849,8 +1867,130 @@ TEST_F(TestValuesWriterInt32Type, AvoidCompressedInDataPageV2) {
verify_only_one_uncompressed_page(/*total_num_values=*/1);
}
}
-
#endif
+// Test writing and reading geometry columns
+class TestGeometryValuesWriter : public TestPrimitiveWriter<ByteArrayType> {
+ public:
+ void SetUpSchema(Repetition::type repetition, int num_columns) override {
+ std::vector<schema::NodePtr> fields;
+
+ for (int i = 0; i < num_columns; ++i) {
+ std::string name = TestColumnName(i);
+ std::shared_ptr<const LogicalType> logical_type =
+ GeometryLogicalType::Make("srid:1234");
+ fields.push_back(schema::PrimitiveNode::Make(name, repetition, logical_type,
+ ByteArrayType::type_num));
+ }
+ node_ = schema::GroupNode::Make("schema", Repetition::REQUIRED, fields);
+ schema_.Init(node_);
+ }
+
+ void GenerateData(int64_t num_values, uint32_t seed = 0) {
+ values_.resize(num_values);
+
+ buffer_.resize(num_values * kWkbPointXYSize);
+ uint8_t* ptr = buffer_.data();
+ for (int k = 0; k < num_values; k++) {
+ std::string item = test::MakeWKBPoint(
+ {static_cast<double>(k), static_cast<double>(k + 1)}, false, false);
+ std::memcpy(ptr, item.data(), item.size());
+ values_[k].len = kWkbPointXYSize;
+ values_[k].ptr = ptr;
+ ptr += kWkbPointXYSize;
+ }
+
+ values_ptr_ = values_.data();
+ }
+};
+
+TEST_F(TestGeometryValuesWriter, TestWriteAndRead) {
+ this->SetUpSchema(Repetition::REQUIRED, 1);
+ this->GenerateData(SMALL_SIZE);
+ size_t num_values = this->values_.size();
+ auto writer = this->BuildWriter(num_values, ColumnProperties());
+ writer->WriteBatch(this->values_.size(), nullptr, nullptr, this->values_.data());
+
+ writer->Close();
+ this->ReadColumn();
+ for (size_t i = 0; i < num_values; i++) {
+ const ByteArray& value = this->values_out_[i];
+ auto xy = GetWKBPointCoordinateXY(value);
+ EXPECT_TRUE(xy.has_value());
+ auto expected_x = static_cast<double>(i);
+ auto expected_y = static_cast<double>(i + 1);
+ EXPECT_EQ(*xy, (std::pair<double, double>(expected_x, expected_y)));
+ }
+
+ ASSERT_TRUE(metadata_accessor()->is_geo_stats_set());
+ std::shared_ptr<GeoStatistics> geospatial_statistics = metadata_geo_stats();
Review Comment:
nit: check that regular statistics is disabled due to unknown sort order.
##########
cpp/src/parquet/test_util.cc:
##########
@@ -194,5 +194,84 @@ void prefixed_random_byte_array(int n, uint32_t seed, uint8_t* buf, int len, FLB
}
}
+namespace {
+
+uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z, bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+} // namespace
+
+std::string MakeWKBPoint(const std::vector<double>& xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes =
+ kWkbPointXYSize + (has_z ? sizeof(double) : 0) + (has_m ? sizeof(double) : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::kPoint, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
+
+ return wkb;
+}
+
+std::optional<std::pair<double, double>> GetWKBPointCoordinateXY(const ByteArray& value) {
+ if (value.len != kWkbPointXYSize) {
+ return std::nullopt;
+ }
+
+ if (value.ptr[0] != kWkbNativeEndianness) {
+ return std::nullopt;
+ }
+
+ uint32_t expected_geom_type =
+ GeometryTypeToWKB(geometry::GeometryType::kPoint, false, false);
Review Comment:
```suggestion
GeometryTypeToWKB(geometry::GeometryType::kPoint, /*has_z=*/false, /*has_m=*/false);
```
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
Review Comment:
Same as my question above: if we don't support wraparound, is the bbox still useful?
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,9 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geospatial_statistics.cc
+ geospatial_util_internal.cc
+ geospatial_util_internal_json.cc
Review Comment:
```suggestion
geospatial_util_json_internal.cc
```
Please rename the header file as well to avoid being installed publicly by accident.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2068145594
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
Review Comment:
I've created https://github.com/apache/arrow/issues/46270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1958667390
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
Review Comment:
I've added documentation to this header/implementation! I'm also happy to add more detail anywhere...the existing internals have pretty much zero documentation, so I went somewhere between that and full-on user-facing documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975806009
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
Review Comment:
I added a comment here! It's the WKB geometry type integer for `POINT ZM`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975905554
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
Review Comment:
I made a few changes here to make this slightly nicer (`std::vector` lets you write `MakeWKBPoint({1, 2}, ...`). I'm happy to update anything about this if there's something else that would make it better!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970039006
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status = bounder_.ReadGeometry(item.ptr, item.len);
+ if (!status.ok()) {
Review Comment:
We could go either way on that...I thought it might be annoying to fail for the case of WKB that we fail to parse, which at the moment just results in missing statistics for that row group. I may have missed it, but I couldn't find a place where we validate UUIDs or JSON (on the other hand, we don't generate statistics for those I don't think). I'm happy to do either 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974518535
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1113,12 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// Write GEOMETRY and GEOGRAPHY logical types where possible.
Review Comment:
I removed this...this use case was that this is a fairly new type and so a mechanism to opt-out might be nice. However, there are two other ways to opt out as well (don't register or unregister the geoarrow extension type or convert the relevant column to the storage type).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974528816
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
Review Comment:
I removed this for now at Antoine's suggestion...we can add it back if we need it for something? It was added initially to parallel `Statistics`, which I think returns a `shared_ptr` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974527892
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
Review Comment:
I updated this comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974519716
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +322,15 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geometry_stats_set() const {
+ DCHECK(writer_version_ != nullptr);
Review Comment:
I kept it to have it parallel all the things that happen to `stats`, but I don't have strong feelings either way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2023081154
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
I just double checked that there shouldn't be any issue if we link parquet with rapidjson. If removing this condition makes it easier to maintain the code, let's do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016160370
##########
python/pyarrow/parquet/core.py:
##########
@@ -254,6 +254,9 @@ class ParquetFile:
it will be parsed as an URI to determine the filesystem.
page_checksum_verification : bool, default False
If True, verify the checksum for each page read from the file.
+ arrow_extensions_enabled : bool, default False
+ If True, read Parquet logical types as Arrow Extension Types where possible,
+ (e.g., JSON arrow.json).
Review Comment:
```suggestion
If True, read Parquet logical types as Arrow extension types where possible,
(e.g., read JSON as the canonical `arrow.json` extension type).
```
##########
python/pyarrow/parquet/core.py:
##########
@@ -1264,6 +1268,9 @@ class ParquetDataset:
sufficient for most Parquet files.
page_checksum_verification : bool, default False
If True, verify the page checksum for each page read from the file.
+arrow_extensions_enabled : bool, default False
+ If True, read Parquet logical types as Arrow Extension Types where possible,
Review Comment:
(same)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017480167
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
+
+ /// \brief Returns true if any non-NaN Y values were encountered or false otherwise
+ bool has_y() const;
+
+ /// \brief Returns true if any non-NaN Z values were encountered or false otherwise
+ bool has_z() const;
+
+ /// \brief Returns true if any non-NaN M values were encountered or false otherwise
+ bool has_m() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> geometry_types() const;
Review Comment:
Internally we accumulate using a set and return a sorted copy...generally returning a copy here allows those things to stay more separated I think if one of the two halves of that needs to change. This list typically contains exactly one element and normally would have less than 4 (the maximum that would ever be returned is 28 although that would be incredibly rare outside test data). I'm happy to update this if it's a sticking point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017528679
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
Review Comment:
I can if I make the private method `MergeGeometryInternal(WKBBuffer* src, bool record_wkb_type);` (and one other one) be `MergeGeometryInternal(void* src, bool record_wkb_type);`. Is it worth it? (Or is there a better way?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008353869
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1152,13 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// \brief Set the GeoCrsContext with which coordinate reference systems should be
+ /// interpreted for GEOMETRY and GEOGRAPHY types
+ Builder* set_geospatial_crs_context(std::shared_ptr<GeoCrsContext> geo_crs_context) {
+ geo_crs_context_ = geo_crs_context;
Review Comment:
Removed!
##########
cpp/src/parquet/properties.h:
##########
@@ -69,6 +69,45 @@ constexpr int32_t kDefaultThriftContainerSizeLimit = 1000 * 1000;
// PARQUET-978: Minimize footer reads by reading 64 KB from the end of the file
constexpr int64_t kDefaultFooterReadSize = 64 * 1024;
+/// \brief Convert Coordinate Reference System values for GEOMETRY and GEOGRAPHY logical
+/// types
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoCrsContext {
+ public:
+ virtual ~GeoCrsContext() = default;
+
+ /// \brief Remove any previously saved PROJJSON CRS fields
+ virtual void Clear() = 0;
+
+ /// \brief Given a coordinate reference system value and encoding from GeoArrow
+ /// extension metadata, return the value that should be placed in the
+ /// LogicalType::Geography|Geometry(crs=) field
+ ///
+ /// For PROJJSON Crses (the most common way coordinate reference systems arrive
+ /// in Arrow), the Parquet specification forces us to write them to the file
+ /// metadata. The default GeoCrsContext will record such values and return the required
+ /// string that can be placed into the 'crs' of the Geometry or Geography logical
+ /// type.
+ virtual std::string GetParquetCrs(std::string crs_value,
Review Comment:
Removed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007889283
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
Review Comment:
I wasn't able to get this to work in mingw (but am open to suggestions if I'm missing something obvious!)
```
In file included from D:/a/arrow/arrow/cpp/src/parquet/types.cc:25:
D:/a/arrow/arrow/cpp/src/arrow/util/checked_cast.h:36:10: error: cannot cast 'const parquet::LogicalType::Impl' to 'const parquet::LogicalType::Impl::Geometry &' via virtual base 'parquet::LogicalType::Impl'
36 | return static_cast<OutputType>(value);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
D:/a/arrow/arrow/cpp/src/parquet/types.cc:1726:10: note: in instantiation of function template specialization 'arrow::internal::checked_cast<const parquet::LogicalType::Impl::Geometry &, const parquet::LogicalType::Impl &>' requested here
1726 | return checked_cast<const LogicalType::Impl::Geometry&>(*impl_).crs();
| ^
In file included from D:/a/arrow/arrow/cpp/src/parquet/types.cc:25:
D:/a/arrow/arrow/cpp/src/arrow/util/checked_cast.h:36:10: error: cannot cast 'const parquet::LogicalType::Impl' to 'const parquet::LogicalType::Impl::Geography &' via virtual base 'parquet::LogicalType::Impl'
36 | return static_cast<OutputType>(value);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
D:/a/arrow/arrow/cpp/src/parquet/types.cc:1830:10: note: in instantiation of function template specialization 'arrow::internal::checked_cast<const parquet::LogicalType::Impl::Geography &, const parquet::LogicalType::Impl &>' requested here
1830 | return checked_cast<const LogicalType::Impl::Geography&>(*impl_).crs();
| ^
2 errors generated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007983781
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007978636
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
Review Comment:
I removed the `null_count` for all the methods here (this was leftover from replicating the statistics interface)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059538434
##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -1090,6 +1094,49 @@ TEST_F(TestConvertParquetSchema, ParquetSchemaArrowUuidExtension) {
}
}
+TEST_F(TestConvertParquetSchema, ParquetSchemaGeoArrowExtensions) {
+ std::vector<NodePtr> parquet_fields;
+ parquet_fields.push_back(PrimitiveNode::Make("geometry", Repetition::OPTIONAL,
+ LogicalType::Geometry(),
+ ParquetType::BYTE_ARRAY));
+ parquet_fields.push_back(PrimitiveNode::Make("geography", Repetition::OPTIONAL,
+ LogicalType::Geography(),
+ ParquetType::BYTE_ARRAY));
+
+ {
+ // Parquet file does not contain Arrow schema.
+ // By default, both fields should be treated as binary() fields in Arrow.
+ auto arrow_schema = ::arrow::schema({::arrow::field("geometry", BINARY, true),
+ ::arrow::field("geography", BINARY, true)});
+ std::shared_ptr<KeyValueMetadata> metadata{};
+ ASSERT_OK(ConvertSchema(parquet_fields, metadata));
+ CheckFlatSchema(arrow_schema);
+ }
+
+ {
+ // Parquet file does not contain Arrow schema.
+ // If Arrow extensions are enabled and extensions are registered,
+ // fields will be interpreted as geoarrow_wkb(binary()) extension fields.
+ ::arrow::ExtensionTypeGuard guard(test::geoarrow_wkb());
+
+ ArrowReaderProperties props;
+ props.set_arrow_extensions_enabled(true);
+ auto arrow_schema = ::arrow::schema(
+ {::arrow::field(
+ "geometry",
+ test::geoarrow_wkb(R"({"crs": "OGC:CRS84", "crs_type": "authority_code"})"),
+ true),
+ ::arrow::field(
+ "geography",
+ test::geoarrow_wkb(
+ R"({"crs": "OGC:CRS84", "crs_type": "authority_code", "edges": "spherical"})"),
+ true)});
+ std::shared_ptr<KeyValueMetadata> metadata{};
+ ASSERT_OK(ConvertSchema(parquet_fields, metadata, props));
+ CheckFlatSchema(arrow_schema);
+ }
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #45459:
URL: https://github.com/apache/arrow/pull/45459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969517360
##########
python/pyarrow/_parquet.pyx:
##########
@@ -440,6 +537,20 @@ cdef class ColumnChunkMetaData(_Weakrefable):
statistics.init(self.metadata.statistics(), self)
return statistics
+ @property
+ def is_geometry_stats_set(self):
+ """Whether or not geometry statistics are present in metadata (bool)."""
+ return self.metadata.is_geometry_stats_set()
+
+ @property
+ def geospatial_statistics(self):
Review Comment:
I (hopefully) updated everything to "geospatial"!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1971325954
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ double xyzm2[] = {-30, -31, -32, -33};
+ std::string xyzm_wkb2 = test::MakeWKBPoint(xyzm2, true, true);
+ ByteArray item2{static_cast<uint32_t>(xyzm_wkb2.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb2.data())};
+
+ // A non-null value that shouldn't be skipped
+ double xyzm3[] = {30, 31, 32, 33};
+ std::string xyzm_wkb3 = test::MakeWKBPoint(xyzm3, true, true);
+ ByteArray item3{static_cast<uint32_t>(xyzm_wkb3.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb3.data())};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeospatialStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_EQ(stats_spaced.GetXMin(), 10);
+ EXPECT_EQ(stats_spaced.GetXMax(), 30);
+ EXPECT_EQ(stats_spaced.GetYMin(), 11);
+ EXPECT_EQ(stats_spaced.GetYMax(), 31);
+ EXPECT_EQ(stats_spaced.GetZMin(), 12);
+ EXPECT_EQ(stats_spaced.GetZMax(), 32);
+ EXPECT_EQ(stats_spaced.GetMMin(), 13);
+ EXPECT_EQ(stats_spaced.GetMMax(), 33);
+ EXPECT_THAT(stats_spaced.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ // Check merge
+ stats.Merge(stats_spaced);
+ EXPECT_TRUE(stats.Equals(stats_spaced));
+
+ // Check ingest of invalid WKB
+ ByteArray invalid;
+ stats.Update(&invalid, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // This should be true even after ingesting more values
+ stats.Update(&item0, 1, 0);
+ EXPECT_FALSE(stats.is_valid());
+ EXPECT_FALSE(stats.Encode().is_set());
+
+ // And should cause other statistics to become invalid when merged with them
+ stats_spaced.Merge(stats);
+ EXPECT_FALSE(stats_spaced.is_valid());
+ EXPECT_FALSE(stats_spaced.Encode().is_set());
+}
+
+TEST(TestGeospatialStatistics, TestUpdateArray) {
+ // Build WKB array with a null from POINT (0 1)...POINT (14, 15)
+ ::arrow::BinaryBuilder builder;
+ std::array<char, test::kWkbPointSize> item;
+ for (int k = 0; k < 10; k++) {
+ test::GenerateWKBPoint(reinterpret_cast<uint8_t*>(item.data()), k, k + 1);
+ ASSERT_OK(builder.AppendValues({std::string(item.data(), item.size())}));
Review Comment:
Ah, it's `Append()` 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974517390
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
+ return R"("crs": ")" + crs.substr(srid_prefix.size()) + R"(", "crs_type": "srid")";
+ } else if (crs.rfind(projjson_prefix, 0) == 0) {
+ std::string metadata_field = crs.substr(projjson_prefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ } else {
+ // Pass on the value of the field so the user can sort this out if needed
+ return R"("crs": )" + metadata_field + R"(, "crs_type": "projjson")";
+ }
+ } else {
+ return Status::Invalid("Can't convert invalid Parquet CRS string to GeoArrow: ", crs);
Review Comment:
I lost track of how strict the wording was in the final spec change...it seems it's just recommended and not required. I think it's a good idea to pass any other string on to GeoArrow verbatim 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016787856
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
Rapidjson linked by Arrow conflicts with the one that we use internally :(
That said, I won't object to it if there is a consensus to make Rapidjson as a required dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2015616569
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
Review Comment:
For NaNs in xmin, ymin, xmax, and ymax, I can't think of a way to make use of statistics that were set to NaN to skip a column chunk; however, the normal predicate pushdown I'm envisioning (a 2D rectangle) can still be done if Z or M dimensions were NaN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016078024
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,9 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geospatial_statistics.cc
Review Comment:
I don't remember if it was already discussed or not, but do we want to put these files in a `geospatial` subdirectory? In the future we might want to expose more utilities there to deal with geo data (or perhaps not?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016111540
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const {
Review Comment:
Only performance-critical methods should be inlined in the header, others should have their definitions in the `.cc` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016115497
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
Review Comment:
Can be removed once `ToString` is moved out of the header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016175373
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
If we mandate ARROW_JSON for Parquet, then we don't need these conditional inclusions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007969246
##########
cpp/src/parquet/arrow/writer.cc:
##########
@@ -526,9 +526,17 @@ Status FileWriter::Make(::arrow::MemoryPool* pool,
Status GetSchemaMetadata(const ::arrow::Schema& schema, ::arrow::MemoryPool* pool,
const ArrowWriterProperties& properties,
std::shared_ptr<const KeyValueMetadata>* out) {
- if (!properties.store_schema()) {
+ if (!properties.store_schema() &&
+ !properties.geo_crs_context()->HasProjjsonCrsFields()) {
Review Comment:
I reverted this change (agreed that it was hard to figure out exactly what would happen for the various combinations of store_schema/HasProjjsonCrsFields())
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008353546
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+namespace {
+
+::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMin);
+ auto max_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMax);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMin);
+ auto max_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMax);
+
+ if (geometry_type_component < min_geometry_type_value ||
+ geometry_type_component > max_geometry_type_value ||
+ dimensions_component < min_dimension_value ||
+ dimensions_component > max_dimension_value) {
+ return ::arrow::Status::SerializationError("Invalid WKB geometry type: ",
+ wkb_geometry_type);
+ }
+
+ GeometryTypeAndDimensions out{static_cast<GeometryType>(geometry_type_component),
+ static_cast<Dimensions>(dimensions_component)};
+ return out;
+}
+
+} // namespace
+
+std::vector<int32_t> WKBGeometryBounder::GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometry(std::string_view bytes_wkb) {
+ WKBBuffer src{reinterpret_cast<const uint8_t*>(bytes_wkb.data()),
+ static_cast<int64_t>(bytes_wkb.size())};
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(&src, /*record_wkb_type=*/true));
+ if (src.size() != 0) {
+ return ::arrow::Status::SerializationError(
+ "Exepcted zero bytes after consuming WKB but got ", src.size());
+ }
+
+ return ::arrow::Status::OK();
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometryInternal(WKBBuffer* src,
+ bool record_wkb_type) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
+ ParseGeometryType(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type_and_dimensions.first) {
+ case GeometryType::kPoint:
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, 1, swap));
+ break;
+
+ case GeometryType::kLinestring: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ break;
+ }
+ case GeometryType::kPolygon: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::kMultiPoint:
+ case GeometryType::kMultiLinestring:
+ case GeometryType::kMultiPolygon:
+ case GeometryType::kGeometryCollection: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(src, /*record_wkb_type*/ false));
Review Comment:
I added a note here...this is because WKB contains recursively nested things (whose geometry type we don't want in the stats...we just want the top-level one).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008350519
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kWKBValueMin = 0,
+ kWKBValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kWKBValueMin = 1,
+ kWKBValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X and Y
+/// dimensions; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status ReadGeometry(std::string_view bytes_wkb);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008350078
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
Review Comment:
I can be convinced either way...my preference is to silently not write statistics for invalid input because Parquet doesn't validiate UUIDs or JSON, either. I think it's OK to error, though, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2744994569
Ok! Thank you for taking another look! I know this is a big PR 😬
In addition to the smaller comments, the two high-level changes here are:
- I removed the `GeoCrsContext` from the `ArrowWriterProperties`. I think there are a lot of things to consider with that particular change and I am not sure this PR is the place to do them. As is, this PR does a direct mapping from the GeoArrow `"crs": ...` to the Parquet logical type `crs`. This allows a high-level producer to implement the recommendation in the format (by writing a GeoArrow CRS of `projjson:some_field` and writing a schema field that contains the appropriate value) but will by default write whatever was in the GeoArrow crs into the Parquet crs. The reverse direction (Parquet -> GeoArrow) does the same thing but does check for the two recommended cases (i.e., it checks in the schema metadata for `some_field` if it sees `projjson:some_field`). This aligns with verbal feedback from @jorisvandenbossche (but I'll let him speak for himself if he has the bandwidth to review this!).
- I made the NaN handling explicit/added tests. NaNs in coordinates are not supposed to happen (except for POINT EMPTY) and other bounding implementations do not have consistent behaviour here. I tried to align the behaviour with what I observed from this Parquet implementation (i.e., NaNs are not included in min/max stats, and if NaNs are read from Thrift they are treated as if no statistics were written). The "treating as if no statistics are written" is not quite as easy to map here because our geo stats contain information about more than one min/max pair, but I at least made sure that nans stay NaNs if they show up in Thrift.
I updated the example in the PR Description as well (some of the comments were out of date). One should be able to swap in an arbitrary GeoPandas GeoDataFrame to give this an interactive spin based on that example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017860210
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
Review Comment:
Done!
##########
python/pyarrow/parquet/core.py:
##########
@@ -1264,6 +1268,9 @@ class ParquetDataset:
sufficient for most Parquet files.
page_checksum_verification : bool, default False
If True, verify the page checksum for each page read from the file.
+arrow_extensions_enabled : bool, default False
+ If True, read Parquet logical types as Arrow Extension Types where possible,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2017866390
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1658,209 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+namespace {
+void WriteCrsKeyAndValue(const std::string& crs, std::ostream& json) {
+ // There is no restriction on the crs value here, and it may contain quotes
+ // or backslashes that would result in invalid JSON if unescaped.
+ json << R"(, "crs": ")";
+ for (char c : crs) {
+ switch (c) {
+ case '"':
+ json << R"(\")";
+ break;
+ case '\\':
+ json << R"(\\)";
+ break;
+ default:
Review Comment:
My solution here is a little clunky...there may be a better way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975810183
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 10);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 11);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 12);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 13);
+ EXPECT_THAT(stats.GetGeometryTypes(), ::testing::ElementsAre(3001));
+
+ double xyzm1[] = {20, 21, 22, 23};
+ std::string xyzm_wkb1 = test::MakeWKBPoint(xyzm1, true, true);
+ ByteArray item1{static_cast<uint32_t>(xyzm_wkb1.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb1.data())};
+
+ stats.Update(&item1, 1, 0);
+ EXPECT_EQ(stats.GetXMin(), 10);
+ EXPECT_EQ(stats.GetXMax(), 20);
+ EXPECT_EQ(stats.GetYMin(), 11);
+ EXPECT_EQ(stats.GetYMax(), 21);
+ EXPECT_EQ(stats.GetZMin(), 12);
+ EXPECT_EQ(stats.GetZMax(), 22);
+ EXPECT_EQ(stats.GetMMin(), 13);
+ EXPECT_EQ(stats.GetMMax(), 23);
Review Comment:
I added `lower_bound()` and `upper_bound()` and check equality with `ElementsAre()` now, which is much nicer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975815872
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
Review Comment:
I updated all of these!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975888826
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975789391
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -21,6 +21,12 @@
#include <string>
#include <vector>
+// TODO(paleolimbot): Remove once example files are generated
Review Comment:
It's done now! I will summarize this in a top-level comment shortly, but I basically moved any JSON or CRS-related code to be in the same place and I'm totally game to remove it and handle the discussion about how it should be done in a separate PR. In the meantime, at least the state of this PR now reflects what was used to write the example files (i.e., the current state is that it's fully functional/useful for writing arbitrary things from GeoPandas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975902486
##########
cpp/src/parquet/reader_test.cc:
##########
@@ -1857,4 +1863,172 @@ TEST(PageIndexReaderTest, ReadFileWithoutPageIndex) {
ASSERT_EQ(nullptr, row_group_index_reader);
}
+class TestGeometryLogicalType : public ::testing::Test {
+ public:
+ const int kNumRows = 1000;
+
+ void WriteTestData(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ // Make schema
+ schema::NodeVector fields;
+ fields.push_back(PrimitiveNode::Make("g", Repetition::REQUIRED,
+ GeometryLogicalType::Make("srid:1234"),
+ Type::BYTE_ARRAY));
+ auto schema = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ // Write small batches and small data pages
+ auto writer_props_builder = WriterProperties::Builder();
+ writer_props_builder.write_batch_size(64)->data_pagesize(128)->data_page_version(
+ data_page_version);
+ if (enable_write_page_index) {
+ writer_props_builder.enable_write_page_index();
+ }
+
+ std::shared_ptr<WriterProperties> writer_props = writer_props_builder.build();
+
+ ASSERT_OK_AND_ASSIGN(auto out_file, ::arrow::io::BufferOutputStream::Create());
+ std::shared_ptr<ParquetFileWriter> file_writer =
+ ParquetFileWriter::Open(out_file, schema, writer_props);
+ RowGroupWriter* rg_writer = file_writer->AppendRowGroup();
+
+ // write WKB points to columns
+ auto* writer = static_cast<ByteArrayWriter*>(rg_writer->NextColumn());
+ if (!write_arrow) {
+ WriteTestDataUsingWriteBatch(writer);
+ } else {
+ WriteTestDataUsingWriteArrow(writer);
+ }
+
+ rg_writer->Close();
+ file_writer->Close();
+
+ ASSERT_OK_AND_ASSIGN(file_buf, out_file->Finish());
+ }
+
+ void WriteTestDataUsingWriteBatch(ByteArrayWriter* writer) {
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ std::vector<ByteArray> values(kNumRows);
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ values[k].len = test::kWkbPointSize;
+ values[k].ptr = ptr;
+ ptr += test::kWkbPointSize;
+ }
+ writer->WriteBatch(kNumRows, nullptr, nullptr, values.data());
+ }
+
+ void WriteTestDataUsingWriteArrow(ByteArrayWriter* writer) {
+ ::arrow::BinaryBuilder builder;
+ std::vector<uint8_t> buffer(test::kWkbPointSize * kNumRows);
+ uint8_t* ptr = buffer.data();
+ for (int k = 0; k < kNumRows; k++) {
+ test::GenerateWKBPoint(ptr, k, k + 1);
+ ASSERT_OK(builder.Append(ptr, test::kWkbPointSize));
+ ptr += test::kWkbPointSize;
+ }
+ std::shared_ptr<::arrow::BinaryArray> array;
+ ASSERT_OK(builder.Finish(&array));
+
+ std::shared_ptr<ArrowWriterProperties> properties =
+ ArrowWriterProperties::Builder().build();
+ MemoryPool* pool = ::arrow::default_memory_pool();
+ auto ctx = std::make_unique<ArrowWriteContext>(pool, properties.get());
+ ASSERT_OK(writer->WriteArrow(nullptr, nullptr, kNumRows, *array, ctx.get(), true));
+ }
+
+ void TestWriteAndRead(ParquetDataPageVersion data_page_version,
+ bool enable_write_page_index, bool write_arrow) {
+ WriteTestData(data_page_version, enable_write_page_index, write_arrow);
+
+ auto in_file = std::make_shared<::arrow::io::BufferReader>(file_buf);
+
+ ReaderProperties reader_props;
+ reader_props.enable_buffered_stream();
+ reader_props.set_buffer_size(64);
+ auto file_reader = ParquetFileReader::Open(in_file, reader_props);
+
+ // Check that the geometry statistics are correctly written and read
+ auto metadata = file_reader->metadata();
+ auto page_index_reader = file_reader->GetPageIndexReader();
+ int num_row_groups = metadata->num_row_groups();
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group_metadata = metadata->RowGroup(i);
+ auto column_chunk_metadata = row_group_metadata->ColumnChunk(0);
+ auto geometry_stats = column_chunk_metadata->geometry_statistics();
+ CheckGeospatialStatistics(geometry_stats);
+ }
+
+ // Check the geometry values
+ int64_t total_values_read = 0;
+ for (int i = 0; i < num_row_groups; i++) {
+ auto row_group = file_reader->RowGroup(i);
+ std::shared_ptr<ByteArrayReader> reader =
+ std::static_pointer_cast<ByteArrayReader>(row_group->Column(0));
+ while (reader->HasNext()) {
+ std::vector<ByteArray> out(kNumRows);
+ int64_t values_read = 0;
+ int64_t levels_read =
+ reader->ReadBatch(kNumRows, nullptr, nullptr, out.data(), &values_read);
+ ASSERT_GE(levels_read, 1);
+ ASSERT_GE(values_read, 1);
+
+ // Check the batch
+ for (int64_t i = 0; i < values_read; i++) {
+ const ByteArray& value = out[i];
+ double x = 0;
+ double y = 0;
+ EXPECT_TRUE(test::GetWKBPointCoordinate(value, &x, &y));
+ auto expected_x = static_cast<double>(i + total_values_read);
+ auto expected_y = static_cast<double>(i + 1 + total_values_read);
+ EXPECT_DOUBLE_EQ(expected_x, x);
+ EXPECT_DOUBLE_EQ(expected_y, y);
+ }
+
+ total_values_read += values_read;
+ }
+ }
+ EXPECT_EQ(kNumRows, total_values_read);
+ }
+
+ void CheckGeospatialStatistics(std::shared_ptr<GeospatialStatistics> geom_stats) {
+ ASSERT_TRUE(geom_stats != nullptr);
+ std::vector<int32_t> geospatial_types = geom_stats->GetGeometryTypes();
+ EXPECT_EQ(1, geospatial_types.size());
+ EXPECT_EQ(1, geospatial_types[0]);
+ EXPECT_GE(geom_stats->GetXMin(), 0);
+ EXPECT_GT(geom_stats->GetXMax(), geom_stats->GetXMin());
+ EXPECT_GT(geom_stats->GetYMin(), 0);
+ EXPECT_GT(geom_stats->GetYMax(), geom_stats->GetYMin());
Review Comment:
I updated these with variable names to show what's going on!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975909166
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
Review Comment:
All the casts in this file use this idiom. I don't mind updating the ones I added but I do feel caught in a number of places between consistency and current standards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975903356
##########
cpp/src/parquet/test_util.h:
##########
@@ -830,5 +833,124 @@ inline void GenerateData<FLBA>(int num_values, FLBA* out, std::vector<uint8_t>*
random_fixed_byte_array(num_values, 0, heap->data(), kGenerateDataFLBALength, out);
}
+// ----------------------------------------------------------------------
+// Test utility functions for geometry
+
+#if defined(ARROW_LITTLE_ENDIAN)
+static constexpr uint8_t kWkbNativeEndianness = 0x01;
+#else
+static constexpr uint8_t kWkbNativeEndianness = 0x00;
+#endif
+
+static uint32_t GeometryTypeToWKB(geometry::GeometryType geometry_type, bool has_z,
+ bool has_m) {
+ auto wkb_geom_type = static_cast<uint32_t>(geometry_type);
+
+ if (has_z) {
+ wkb_geom_type += 1000;
+ }
+
+ if (has_m) {
+ wkb_geom_type += 2000;
+ }
+
+ return wkb_geom_type;
+}
+
+static inline std::string MakeWKBPoint(const double* xyzm, bool has_z, bool has_m) {
+ // 1:endianness + 4:type + 8:x + 8:y
+ int num_bytes = 21 + (has_z ? 8 : 0) + (has_m ? 8 : 0);
+ std::string wkb(num_bytes, 0);
+ char* ptr = wkb.data();
+
+ ptr[0] = kWkbNativeEndianness;
+ uint32_t geom_type = GeometryTypeToWKB(geometry::GeometryType::POINT, has_z, has_m);
+ std::memcpy(&ptr[1], &geom_type, 4);
+ std::memcpy(&ptr[5], &xyzm[0], 8);
+ std::memcpy(&ptr[13], &xyzm[1], 8);
+ ptr += 21;
+
+ if (has_z) {
+ std::memcpy(ptr, &xyzm[2], 8);
+ ptr += 8;
+ }
+ if (has_m) {
+ std::memcpy(ptr, &xyzm[3], 8);
+ }
+
+ return wkb;
+}
+
+static constexpr int kWkbPointSize = 21; // 1:endianness + 4:type + 8:x + 8:y
+
+inline void GenerateWKBPoint(uint8_t* ptr, double x, double y) {
+ double xyzm[] = {x, y, geometry::kInf, geometry::kInf};
+ std::string wkb = MakeWKBPoint(xyzm, false, false);
+ std::memcpy(ptr, wkb.data(), kWkbPointSize);
Review Comment:
I made this variable name more clear and added a comment (an XY point will always be 21 bytes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-45522]: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2655539648
@wgtmac This is ready for a first look! I've noted a few things about scope that could be dropped from this PR to the Description...I'm happy to do this in any order you'd like. Let me know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953935283
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(std::array<double, 4> coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ double min[4];
+ double max[4];
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return std::memcmp(lhs.min, rhs.min, sizeof(lhs.min)) == 0 &&
+ std::memcmp(lhs.max, rhs.max, sizeof(lhs.max)) == 0;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src, bool record_wkb_type = true) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type, GeometryType::FromWKB(wkb_geometry_type));
+ ARROW_ASSIGN_OR_RAISE(auto dimensions, Dimensions::FromWKB(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type) {
+ case GeometryType::geometry_type::POINT:
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, 1, swap));
+ break;
+
+ case GeometryType::geometry_type::LINESTRING: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, n_coords, swap));
+ break;
+ }
+ case GeometryType::geometry_type::POLYGON: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(ReadSequence(src, dimensions, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::geometry_type::MULTIPOINT:
+ case GeometryType::geometry_type::MULTILINESTRING:
+ case GeometryType::geometry_type::MULTIPOLYGON:
+ case GeometryType::geometry_type::GEOMETRYCOLLECTION: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometry(src, /*record_wkb_type*/ false));
+ }
+ break;
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ const BoundingBox& Bounds() const { return box_; }
+
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions::dimensions dimensions,
+ uint32_t n_coords, bool swap) {
+ using XY = std::array<double, 2>;
Review Comment:
defining these within a class or struct and commenting them, then using them in other UpdateXYZ methods might make some of the code more readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955187371
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ if (size_ < sizeof(uint32_t)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read 4 bytes from from WKBBuffer with ", size_, " remaining");
+ }
+
+ uint32_t value = ::arrow::util::SafeLoadAs<uint32_t>(data_);
+ data_ += sizeof(uint32_t);
+ size_ -= sizeof(uint32_t);
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ::arrow::bit_util::ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Func>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Func&& func) {
+ if (n_coords == 0) {
+ return ::arrow::Status::OK();
+ }
+
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", total_bytes, " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ::arrow::util::SafeLoadAs<Coord>(data_);
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ::arrow::bit_util::ByteSwap(coord[j]);
+ }
+
+ func(coord);
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ func(::arrow::util::SafeLoadAs<Coord>(data_));
+ data_ += sizeof(Coord);
+ size_ -= sizeof(Coord);
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+};
+
+struct BoundingBox {
+ BoundingBox(const std::array<double, 4>& mins, const std::array<double, 4>& maxes) {
+ std::memcpy(min, mins.data(), sizeof(min));
+ std::memcpy(max, maxes.data(), sizeof(max));
+ }
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(std::array<double, 2> coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(std::array<double, 3> coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(std::array<double, 3> coord) {
Review Comment:
I moved them all to be by reference here (I would be surprised if a compiler didn't inline these calls either way but I'm also not an expert!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953912186
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
+ case 1:
+ return geometry_type::POINT;
+ case 2:
+ return geometry_type::LINESTRING;
+ case 3:
+ return geometry_type::POLYGON;
+ case 4:
+ return geometry_type::MULTIPOINT;
+ case 5:
+ return geometry_type::MULTILINESTRING;
+ case 6:
+ return geometry_type::MULTIPOLYGON;
+ case 7:
+ return geometry_type::GEOMETRYCOLLECTION;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static std::string ToString(geometry_type geometry_type) {
+ switch (geometry_type) {
+ case geometry_type::POINT:
+ return "POINT";
+ case geometry_type::LINESTRING:
+ return "LINESTRING";
+ case geometry_type::POLYGON:
+ return "POLYGON";
+ case geometry_type::MULTIPOINT:
+ return "MULTIPOINT";
+ case geometry_type::MULTILINESTRING:
+ return "MULTILINESTRING";
+ case geometry_type::MULTIPOLYGON:
+ return "MULTIPOLYGON";
+ case geometry_type::GEOMETRYCOLLECTION:
+ return "GEOMETRYCOLLECTION";
+ default:
+ return "";
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() {
+ if (size_ < 1) {
+ throw ParquetException("Can't read 1 byte from empty WKBBuffer");
+ }
+
+ size_ -= 1;
+ return *data_++;
+ }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
Review Comment:
is there a reason all class implementations are in the header? (holdover from templating)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2024523967
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -171,6 +171,9 @@ set(PARQUET_SRCS
exception.cc
file_reader.cc
file_writer.cc
+ geospatial_statistics.cc
Review Comment:
Yes, I think it would be cleaner in a dedicated subdirectory (for example `geospatial/statistics.cc`).
We could do the same for the various `level_` files at some point :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970112764
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
Review Comment:
I'm not sure! Because the zmin/zmax go together I think the code will always look like `if (stats.has_z()) ...`. The only case where X and Y are empty should be if no coordinates were encountered (e.g., all empty or null geometries), in which case the optional-ness of xmin, ymin, xmax, and ymax go together. If we think that `optional` is clearer for the individual values as well I'm happy to add that in!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970101830
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetXMax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
+ /// case, these bounds represent the union of the intervals [ymax, Inf] and [-Inf,
+ /// ymin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetYMin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no Y values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double GetYMax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no Z values were
+ /// encountered. Wrap around bounds are not permitted in the Z dimension.
+ double GetZMax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no M values were
+ /// encountered. Wrap around bounds are not permitted in the M dimension.
+ double GetMMax() const;
+
+ /// \brief Returns true if any Z values were encountered or false otherwise
+ bool HasZ() const;
+
+ /// \brief Returns true if any M values were encountered or false otherwise
+ bool HasM() const;
+
+ /// \brief Return the geometry type codes from the well-known binary encountered
+ ///
+ /// This implementation always returns sorted output with no duplicates.
+ std::vector<int32_t> GetGeometryTypes() const;
Review Comment:
> Is it generated on the fly?
Yes, it's accumulated as an `unordered_set` but it's somewhat less confusing if we return it canonically ordered. That's not a hard requirement if there's a good reason not to do it 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953924369
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
+ enum class geometry_type {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+ };
+
+ static ::arrow::Result<geometry_type> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type % 1000) {
Review Comment:
can 1000 be made a nemonic constant? (is there a pointer to the spec on why 1000?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1953925848
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,395 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+#include "parquet/exception.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+struct Dimensions {
+ enum class dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+ static ::arrow::Result<dimensions> FromWKB(uint32_t wkb_geometry_type) {
+ switch (wkb_geometry_type / 1000) {
+ case 0:
+ return dimensions::XY;
+ case 1:
+ return dimensions::XYZ;
+ case 2:
+ return dimensions::XYM;
+ case 3:
+ return dimensions::XYZM;
+ default:
+ return ::arrow::Status::SerializationError("Invalid wkb_geometry_type: ",
+ wkb_geometry_type);
+ }
+ }
+
+ static uint32_t size(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return 2;
+ case dimensions::XYZ:
+ case dimensions::XYM:
+ return 3;
+ case dimensions::XYZM:
+ return 4;
+ default:
+ return 0;
+ }
+ }
+
+ static std::string ToString(dimensions dims) {
+ switch (dims) {
+ case dimensions::XY:
+ return "XY";
+ case dimensions::XYZ:
+ return "XYZ";
+ case dimensions::XYM:
+ return "XYM";
+ case dimensions::XYZM:
+ return "XYZM";
+ default:
+ return "";
+ }
+ }
+};
+
+struct GeometryType {
Review Comment:
not sure what is standard in Geo Naming, but could this be called Geometry and the nested enum by called `type`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975809536
##########
cpp/src/parquet/geometry_statistics_test.cc:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "geometry_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeospatialStatistics, TestDefaults) {
+ GeospatialStatistics stats;
+ EXPECT_EQ(stats.GetGeometryTypes().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_FALSE(stats.HasZ());
+ EXPECT_FALSE(stats.HasM());
+ EXPECT_EQ(stats.GetXMax() - stats.GetXMin(), -kInf);
+ EXPECT_EQ(stats.GetYMax() - stats.GetYMin(), -kInf);
+ EXPECT_EQ(stats.GetZMax() - stats.GetZMin(), -kInf);
+ EXPECT_EQ(stats.GetMMax() - stats.GetMMin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeospatialStatistics()));
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeospatialStatistics());
+ EXPECT_TRUE(GeospatialStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeospatialStatistics, TestUpdateByteArray) {
+ GeospatialStatistics stats;
+
+ double xyzm0[] = {10, 11, 12, 13};
+ std::string xyzm_wkb0 = test::MakeWKBPoint(xyzm0, true, true);
+ ByteArray item0{static_cast<uint32_t>(xyzm_wkb0.size()),
+ reinterpret_cast<const uint8_t*>(xyzm_wkb0.data())};
+
+ stats.Update(&item0, 1, 0);
Review Comment:
> Why does `stats.Update` take a pointer?
It's taking an array of `ByteArray`, so the pointer is to the first element. This is identical to how `Statistics` works as well.
I added in the parameter comments as well wherever possible!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975815629
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
Review Comment:
I updated all of these!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975883636
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
+ uint32_t wkb_type =
+ static_cast<int>(item.dimensions) * 1000 + static_cast<int>(item.geometry_type);
+ EXPECT_THAT(bounder.GeometryTypes(), ::testing::ElementsAre(::testing::Eq(wkb_type)));
+
+ bounder.Reset();
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+ EXPECT_TRUE(bounder.GeometryTypes().empty());
+}
+
+INSTANTIATE_TEST_SUITE_P(
+ TestGeometryUtil, WKBTestFixture,
+ ::testing::Values(
+ // POINT (30 10)
+ WKBTestCase(GeometryType::POINT, Dimensions::XY,
+ {0x01, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // POINT Z (30 10 40)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZ,
+ {0x01, 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // POINT M (30 10 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYM,
+ {0x01, 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // POINT ZM (30 10 40 300)
+ WKBTestCase(GeometryType::POINT, Dimensions::XYZM,
+ {0x01, 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // LINESTRING (30 10, 10 30, 40 40)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XY,
+ {0x01, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // LINESTRING Z (30 10 40, 10 30 40, 40 40 80)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZ,
+ {0x01, 0xea, 0x03, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // LINESTRING M (30 10 300, 10 30 300, 40 40 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYM,
+ {0x01, 0xd2, 0x07, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0,
+ 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // LINESTRING ZM (30 10 40 300, 10 30 40 300, 40 40 80 1600)
+ WKBTestCase(GeometryType::LINESTRING, Dimensions::XYZM,
+ {0x01, 0xba, 0x0b, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))
+ WKBTestCase(GeometryType::POLYGON, Dimensions::XY,
+ {0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // POLYGON Z ((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZ,
+ {0x01, 0xeb, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // POLYGON M ((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYM,
+ {0x01, 0xd3, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // POLYGON ZM ((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30 10 40
+ // 300))
+ WKBTestCase(
+ GeometryType::POLYGON, Dimensions::XYZM,
+ {0x01, 0xbb, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // MULTIPOINT ((30 10))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XY,
+ {0x01, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // MULTIPOINT Z ((30 10 40))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZ,
+ {0x01, 0xec, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // MULTIPOINT M ((30 10 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYM,
+ {0x01, 0xd4, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // MULTIPOINT ZM ((30 10 40 300))
+ WKBTestCase(GeometryType::MULTIPOINT, Dimensions::XYZM,
+ {0x01, 0xbc, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300}),
+ // MULTILINESTRING ((30 10, 10 30, 40 40))
+ WKBTestCase(GeometryType::MULTILINESTRING, Dimensions::XY,
+ {0x01, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x02,
+ 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 40, 40}),
+ // MULTILINESTRING Z ((30 10 40, 10 30 40, 40 40 80))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZ,
+ {0x01, 0xed, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xea, 0x03, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x40},
+ {10, 10, 40, 40, 40, 80}),
+ // MULTILINESTRING M ((30 10 300, 10 30 300, 40 40 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYM,
+ {0x01, 0xd5, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd2, 0x07, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 300, 40, 40, 1600}),
+ // MULTILINESTRING ZM ((30 10 40 300, 10 30 40 300, 40 40 80 1600))
+ WKBTestCase(
+ GeometryType::MULTILINESTRING, Dimensions::XYZM,
+ {0x01, 0xbd, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xba, 0x0b, 0x00,
+ 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40},
+ {10, 10, 40, 300, 40, 40, 80, 1600}),
+ // MULTIPOLYGON (((30 10, 40 40, 20 40, 10 20, 30 10)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XY,
+ {0x01, 0x06, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {10, 10, 40, 40}),
+ // MULTIPOLYGON Z (((30 10 40, 40 40 80, 20 40 60, 10 20 30, 30 10 40)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYZ,
+ {0x01, 0xee, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xeb, 0x03, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x54, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x4e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {10, 10, 30, 40, 40, 80}),
+ // MULTIPOLYGON M (((30 10 300, 40 40 1600, 20 40 800, 10 20 200, 30 10 300)))
+ WKBTestCase(
+ GeometryType::MULTIPOLYGON, Dimensions::XYM,
+ {0x01, 0xd6, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xd3, 0x07, 0x00,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x89, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x69,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 200, 40, 40, 1600}),
+ // MULTIPOLYGON ZM (((30 10 40 300, 40 40 80 1600, 20 40 60 800, 10 20 30 200, 30
+ // 10 40 300)))
+ WKBTestCase(GeometryType::MULTIPOLYGON, Dimensions::XYZM,
+ {0x01, 0xbe, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0xbb,
+ 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x99, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x44, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4e, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x40, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x24, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x34, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3e, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x69, 0x40, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24,
+ 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {10, 10, 30, 200, 40, 40, 80, 1600}),
+ // GEOMETRYCOLLECTION (POINT (30 10))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XY,
+ {0x01, 0x07, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40},
+ {30, 10, 30, 10}),
+ // GEOMETRYCOLLECTION Z (POINT Z (30 10 40))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZ,
+ {0x01, 0xef, 0x03, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xe9, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40},
+ {30, 10, 40, 30, 10, 40}),
+ // GEOMETRYCOLLECTION M (POINT M (30 10 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYM,
+ {0x01, 0xd7, 0x07, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xd1, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 300, 30, 10, 300}),
+ // GEOMETRYCOLLECTION ZM (POINT ZM (30 10 40 300))
+ WKBTestCase(GeometryType::GEOMETRYCOLLECTION, Dimensions::XYZM,
+ {0x01, 0xbf, 0x0b, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+ 0xb9, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x3e, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x40,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x40, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0xc0, 0x72, 0x40},
+ {30, 10, 40, 300, 30, 10, 40, 300})));
+
+struct MakeWKBPointTestCase {
+ MakeWKBPointTestCase() = default;
+ MakeWKBPointTestCase(const std::vector<double> xyzm, bool has_z, bool has_m)
+ : has_z(has_z), has_m(has_m) {
+ memcpy(this->xyzm, xyzm.data(), sizeof(this->xyzm));
+ }
+
+ double xyzm[4];
Review Comment:
Yes! I believe I updated caught all `double*` or `double[]` variables and converted them to `span` or `std::array`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975881021
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
+
+ GeometryType geometry_type;
+ Dimensions dimensions;
+ std::vector<uint8_t> wkb;
+ BoundingBox box;
+};
+
+std::ostream& operator<<(std::ostream& os, const WKBTestCase& obj) {
+ uint32_t iso_wkb_geometry_type =
+ static_cast<int>(obj.dimensions) * 1000 + static_cast<int>(obj.geometry_type);
+ os << "WKBTestCase<" << iso_wkb_geometry_type << ">";
+ return os;
+}
+
+std::ostream& operator<<(std::ostream& os, const BoundingBox& obj) {
+ os << obj.ToString();
+ return os;
+}
+
+class WKBTestFixture : public ::testing::TestWithParam<WKBTestCase> {
+ protected:
+ WKBTestCase test_case;
+};
+
+TEST_P(WKBTestFixture, TestWKBBounderNonEmpty) {
+ auto item = GetParam();
+
+ WKBGeometryBounder bounder;
+ EXPECT_EQ(bounder.Bounds(), BoundingBox());
+
+ ASSERT_OK(bounder.ReadGeometry(item.wkb.data(), item.wkb.size()));
+
+ EXPECT_EQ(bounder.Bounds(), item.box);
Review Comment:
I added this check!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955570020
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) >= 0; }
+
+ bool has_m() const { return (mmax - mmin) >= 0; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+class PARQUET_EXPORT GeospatialStatistics {
Review Comment:
same comment on docs in this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1956562288
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ if (!is_valid_ || !other.is_valid_) {
+ is_valid_ = false;
+ return;
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ buf.Init(item.ptr, item.len);
+ ::arrow::Status status = bounder_.ReadGeometry(&buf);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ buf.Init(item.ptr, item.len);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(&buf));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ const auto& binary_array = static_cast<const ::arrow::BinaryArray&>(values);
Review Comment:
Good call! I added the LargeBinary support + type check + tests in the arrow writer (but it looks like views aren't supported, or at least I get `Arrow type binary_view cannot be written to Parquet type column descriptor` when I try to test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974527594
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
Review Comment:
I added `has_x()` and `has_y()`, which will both be`false` if everything is null or there have been no values bounded (because all geometries have X and Y`).
`NaN` values are ignored, because an empty POINT is stored (by convention) as all `NaN`. I suppose this has the consequence that something with all NaN values might not be returned if a spatial filter is being pushed down. I personally think this is OK but will check to see what GDAL does for that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974511246
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
+ return R"("crs": ")" + crs.substr(srid_prefix.size()) + R"(", "crs_type": "srid")";
+ } else if (crs.rfind(projjson_prefix, 0) == 0) {
+ std::string metadata_field = crs.substr(projjson_prefix.size());
+ if (metadata && metadata->Contains(metadata_field)) {
+ ARROW_ASSIGN_OR_RAISE(std::string projjson_value, metadata->Get(metadata_field));
+ return R"("crs": )" + projjson_value + R"(, "crs_type": "projjson")";
+ } else {
+ // Pass on the value of the field so the user can sort this out if needed
+ return R"("crs": )" + metadata_field + R"(, "crs_type": "projjson")";
+ }
+ } else {
+ return Status::Invalid("Can't convert invalid Parquet CRS string to GeoArrow: ", crs);
+ }
+}
+
+Result<std::shared_ptr<ArrowType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ // Check if we have a registered GeoArrow type to read into
+ std::shared_ptr<::arrow::ExtensionType> maybe_geoarrow_wkb =
+ ::arrow::GetExtensionType("geoarrow.wkb");
+ if (!maybe_geoarrow_wkb) {
+ return ::arrow::binary();
+ }
+
+ if (logical_type.is_geometry()) {
+ const auto& geospatial_type = checked_cast<const GeometryLogicalType&>(logical_type);
+ ARROW_ASSIGN_OR_RAISE(std::string crs_metadata,
+ MakeGeoArrowCrsMetadata(geospatial_type.crs(), metadata));
+
+ std::string serialized_data = std::string("{") + crs_metadata + "}";
+ return maybe_geoarrow_wkb->Deserialize(::arrow::binary(), serialized_data);
Review Comment:
Yes, the extension type implementation takes the JSON its given and (usually) parses/validates it into some non-serialized structure. (We don't really care what that is here, we just need the `shared_ptr<DataType>` 🙂 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974550456
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
Review Comment:
I added a comment here...I didn't attempt geography bounding here (so geography types are written without statistics for now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2068140399
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,194 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+inline constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+inline constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{false};
+ double xmin{kNaN};
+ double xmax{kNaN};
+ double ymin{kNaN};
+ double ymax{kNaN};
+
+ bool z_bounds_present{false};
+ double zmin{kNaN};
+ double zmax{kNaN};
+
+ bool m_bounds_present{false};
+ double mmin{kNaN};
+ double mmax{kNaN};
+
+ bool geospatial_types_present() const { return !geospatial_types.empty(); }
+ std::vector<int32_t> geospatial_types;
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ ///
+ /// Merging statistics with wraparound X values is not currently supported. Merging
+ /// two GeoStatistics where one or both has a wraparound X range will result in these
+ /// statistics having an X dimension marked as invalid.
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered or if the statistics contain NaN
+ /// for any reason, Encode() will return nullopt to indicate that
+ /// statistics should not be written to thrift.
+ std::optional<EncodedGeoStatistics> Encode() const;
+
+ /// \brief Returns false if invalid WKB was encountered
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief Minimum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be +Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> lower_bound() const;
+
+ /// \brief Maximum values in XYZM order
+ ///
+ /// For dimensions where dimension_valid() is false, the value will be NaN. For
+ /// dimensions where dimension_empty() is true, the value will be -Inf.
+ ///
+ /// For the first dimension (X) only, wraparound bounds apply where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in statistics when reading a Parquet file.
+ std::array<double, kMaxDimensions> upper_bound() const;
+
+ /// \brief Dimension emptiness in XYZM order
+ ///
+ /// True for a given dimension if and only if zero non-NaN values were encountered
+ /// in that dimension and dimension_valid() is true for that dimension.
Review Comment:
@paleolimbot You misunderstood this comment.
If a method named `dimension_empty`, then returning `true` should mean the dimension is "empty" and returning `false` should mean the dimension is not "empty" (regardless of the meaning).
This method does the reverse, can you change it in a followup PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2002351474
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
Review Comment:
If all members are public, what about using a `struct`?
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1658,193 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
+}
+
+std::shared_ptr<const LogicalType> GeometryLogicalType::Make(std::string crs) {
+ auto* logical_type = new GeometryLogicalType();
+ logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
+ return std::shared_ptr<const LogicalType>(logical_type);
Review Comment:
```suggestion
auto logical_type = std::shared_ptr<GeometryLogicalType>(new GeometryLogicalType());
logical_type->impl_.reset(new LogicalType::Impl::Geometry(std::move(crs)));
return logical_type;
```
I know it is for consistency. Is it better to construct it as above?
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
Review Comment:
```suggestion
return !geospatial_types.empty() || (has_x() && has_y());
```
Should it be the above?
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
Review Comment:
Why do we need a `null_count` here?
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
Review Comment:
```suggestion
/// be encountered in files written by other writers.
```
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kWKBValueMin = 0,
+ kWKBValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kWKBValueMin = 1,
+ kWKBValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X and Y
+/// dimensions; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status ReadGeometry(std::string_view bytes_wkb);
Review Comment:
Instead of `ReadXXX`, should we call them `MergeXXX|AddXXX` to be explicit?
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
Review Comment:
Why do we need the default ctor and `Init`? For reuse?
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1227,10 +1239,14 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
// Will be null if not using dictionary, but that's ok
current_dict_encoder_ = dynamic_cast<DictEncoder<DType>*>(current_encoder_.get());
- if (properties->statistics_enabled(descr_->path()) &&
- (SortOrder::UNKNOWN != descr_->sort_order())) {
- page_statistics_ = MakeStatistics<DType>(descr_, allocator_);
- chunk_statistics_ = MakeStatistics<DType>(descr_, allocator_);
+ if (properties->statistics_enabled(descr_->path())) {
+ if (SortOrder::UNKNOWN != descr_->sort_order()) {
Review Comment:
Since geo types have SortOrder::UNKNOWN, this also disables stats like value_count, null_count for them. However, SortOrder only affects the lower/upper bounds. Perhaps we need to add a comment here to improve.
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -1088,6 +1114,10 @@ Status ToParquetSchema(const ::arrow::Schema* arrow_schema,
const WriterProperties& properties,
const ArrowWriterProperties& arrow_properties,
std::shared_ptr<SchemaDescriptor>* out) {
+ // TODO(paleolimbot): I'm wondering if this geo_crs_context is reused when testing
+ // on MINGW, where we get some failures indicating non-empty metadata where it was
+ // expected
+ arrow_properties.geo_crs_context()->Clear();
Review Comment:
It is weird to call `Clear()` on a const reference. Why doing this?
##########
cpp/src/parquet/arrow/writer.cc:
##########
@@ -526,9 +526,17 @@ Status FileWriter::Make(::arrow::MemoryPool* pool,
Status GetSchemaMetadata(const ::arrow::Schema& schema, ::arrow::MemoryPool* pool,
const ArrowWriterProperties& properties,
std::shared_ptr<const KeyValueMetadata>* out) {
- if (!properties.store_schema()) {
+ if (!properties.store_schema() &&
+ !properties.geo_crs_context()->HasProjjsonCrsFields()) {
Review Comment:
Can we enforce `store_schema` to be true (or throw when it is false) when `HasProjjsonCrsFields()` is true? This is easier to maintain the code.
##########
cpp/src/parquet/types.cc:
##########
@@ -1619,6 +1668,204 @@ class LogicalType::Impl::Float16 final : public LogicalType::Impl::Incompatible,
GENERATE_MAKE(Float16)
+class LogicalType::Impl::Geometry final : public LogicalType::Impl::Incompatible,
+ public LogicalType::Impl::SimpleApplicable {
+ public:
+ friend class GeometryLogicalType;
+
+ std::string ToString() const override;
+ std::string ToJSON() const override;
+ format::LogicalType ToThrift() const override;
+ bool Equals(const LogicalType& other) const override;
+
+ const std::string& crs() const { return crs_; }
+
+ private:
+ explicit Geometry(std::string crs)
+ : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
+ LogicalType::Impl::SimpleApplicable(parquet::Type::BYTE_ARRAY),
+ crs_(std::move(crs)) {}
+
+ std::string crs_;
+};
+
+std::string LogicalType::Impl::Geometry::ToString() const {
+ std::stringstream type;
+ type << "Geometry(crs=" << crs_ << ")";
+ return type.str();
+}
+
+std::string LogicalType::Impl::Geometry::ToJSON() const {
+ std::stringstream json;
+ json << R"({"Type": "Geometry")";
+
+ if (!crs_.empty()) {
+ // TODO(paleolimbot): For documented cases the CRS shouldn't contain quotes,
+ // but we should probably escape the value of crs_ for backslash and quotes
+ // to be safe.
+ json << R"(, "crs": ")" << crs_ << R"(")";
+ }
+
+ json << "}";
+ return json.str();
+}
+
+format::LogicalType LogicalType::Impl::Geometry::ToThrift() const {
+ format::LogicalType type;
+ format::GeometryType geometry_type;
+
+ // Canonially export crs of "" as an unset CRS
+ if (!crs_.empty()) {
+ geometry_type.__set_crs(crs_);
+ }
+
+ type.__set_GEOMETRY(geometry_type);
+ return type;
+}
+
+bool LogicalType::Impl::Geometry::Equals(const LogicalType& other) const {
+ if (other.is_geometry()) {
+ const auto& other_geometry = checked_cast<const GeometryLogicalType&>(other);
+ return crs() == other_geometry.crs();
+ } else {
+ return false;
+ }
+}
+
+const std::string& GeometryLogicalType::crs() const {
+ return (dynamic_cast<const LogicalType::Impl::Geometry&>(*impl_)).crs();
Review Comment:
I'm inclined to use `checked_cast` here. Perhaps creating an issue for each kind of inconsistency?
##########
cpp/src/parquet/properties.h:
##########
@@ -69,6 +69,45 @@ constexpr int32_t kDefaultThriftContainerSizeLimit = 1000 * 1000;
// PARQUET-978: Minimize footer reads by reading 64 KB from the end of the file
constexpr int64_t kDefaultFooterReadSize = 64 * 1024;
+/// \brief Convert Coordinate Reference System values for GEOMETRY and GEOGRAPHY logical
+/// types
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoCrsContext {
+ public:
+ virtual ~GeoCrsContext() = default;
+
+ /// \brief Remove any previously saved PROJJSON CRS fields
+ virtual void Clear() = 0;
+
+ /// \brief Given a coordinate reference system value and encoding from GeoArrow
+ /// extension metadata, return the value that should be placed in the
+ /// LogicalType::Geography|Geometry(crs=) field
+ ///
+ /// For PROJJSON Crses (the most common way coordinate reference systems arrive
+ /// in Arrow), the Parquet specification forces us to write them to the file
+ /// metadata. The default GeoCrsContext will record such values and return the required
+ /// string that can be placed into the 'crs' of the Geometry or Geography logical
+ /// type.
+ virtual std::string GetParquetCrs(std::string crs_value,
+ const std::string& crs_encoding) = 0;
+
+ /// \brief Returns true if any converted CRS values were PROJJSON whose values are
+ /// stored in this object and should be written to the file metadata
+ virtual bool HasProjjsonCrsFields() = 0;
+
+ /// \brief Add any stored PROJJSON values to the supplied file KeyValueMetadata
+ ///
+ /// The default implementation will add any PROJJSON values it encountered to the file
+ /// KeyValueMetadata; however, a custom implementation may choose to store these values
+ /// somewhere else and implement this method as a no-op.
+ virtual void AddProjjsonCrsFieldsToFileMetadata(
Review Comment:
```suggestion
virtual void AddProjjsonCrsFieldsToMetadata(
```
It is not necessary to be File.
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,206 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kWKBValueMin = 0,
Review Comment:
Why `kWKBValueMin/kWKBValueMax` but not `kDimValueMin/kDimValueMax` or simply `kValueMin/kValueMax`?
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double get_xmin() const;
Review Comment:
```suggestion
double xmin() const;
```
We don't really use `get_xxx()` in parquet. Same for below.
##########
cpp/src/parquet/geospatial_util_internal_json.h:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/properties.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromGeoArrowJSON(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties);
+
+/// \brief Compute a suitable DataType into which a GEOMETRY or GEOGRAPHY type should be
+/// read
+///
+/// The result of this function depends on whether or not "geoarrow.wkb" has been
+/// registered: if it has, the result will be the registered ExtensionType; if it has not,
+/// the result will be binary().
+::arrow::Result<std::shared_ptr<::arrow::DataType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata);
+
+/// \brief The default GeoCrsContext, which writes any PROJJSON coordinate reference
+/// system values it encounters to the file metadata
+class PARQUET_EXPORT FileGeoCrsContext : public GeoCrsContext {
+ public:
+ FileGeoCrsContext() { Clear(); }
+
+ void Clear() override {
+ projjson_crs_fields_ = ::arrow::KeyValueMetadata::Make({}, {});
+ }
+
+ std::string GetParquetCrs(std::string crs_value,
+ const std::string& crs_encoding) override;
+
+ bool HasProjjsonCrsFields() override { return projjson_crs_fields_->size() > 0; }
+
+ void AddProjjsonCrsFieldsToFileMetadata(::arrow::KeyValueMetadata* metadata) override;
+
+ private:
+ std::shared_ptr<::arrow::KeyValueMetadata> projjson_crs_fields_;
Review Comment:
Why not use a simple std::map or std::unordered_map ?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.ReadGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status =
+ bounder_.ReadGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
Review Comment:
`null_count` is unused
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double get_xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
Review Comment:
Will `xmax` return `Inf`? How do we handle `NaN`? It would be good to document these behaviors as well.
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+namespace {
+
+::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMin);
+ auto max_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMax);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMin);
+ auto max_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMax);
+
+ if (geometry_type_component < min_geometry_type_value ||
+ geometry_type_component > max_geometry_type_value ||
+ dimensions_component < min_dimension_value ||
+ dimensions_component > max_dimension_value) {
+ return ::arrow::Status::SerializationError("Invalid WKB geometry type: ",
+ wkb_geometry_type);
+ }
+
+ GeometryTypeAndDimensions out{static_cast<GeometryType>(geometry_type_component),
+ static_cast<Dimensions>(dimensions_component)};
+ return out;
+}
+
+} // namespace
+
+std::vector<int32_t> WKBGeometryBounder::GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometry(std::string_view bytes_wkb) {
+ WKBBuffer src{reinterpret_cast<const uint8_t*>(bytes_wkb.data()),
+ static_cast<int64_t>(bytes_wkb.size())};
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(&src, /*record_wkb_type=*/true));
+ if (src.size() != 0) {
+ return ::arrow::Status::SerializationError(
+ "Exepcted zero bytes after consuming WKB but got ", src.size());
+ }
+
+ return ::arrow::Status::OK();
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometryInternal(WKBBuffer* src,
+ bool record_wkb_type) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
+ ParseGeometryType(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type_and_dimensions.first) {
+ case GeometryType::kPoint:
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, 1, swap));
+ break;
+
+ case GeometryType::kLinestring: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ break;
+ }
+ case GeometryType::kPolygon: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::kMultiPoint:
+ case GeometryType::kMultiLinestring:
+ case GeometryType::kMultiPolygon:
+ case GeometryType::kGeometryCollection: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(src, /*record_wkb_type*/ false));
Review Comment:
If `record_wkb_type` is false, we are losing track of these encountered geometry types. Does it obviates the spec which requests to clear the list in this case? Why not manually add the GeometryType?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
Review Comment:
Why not `return is_valid == other.is_valid_ && bounder_.GeometryTypes() == other.bounder_.GeometryTypes() && bounder_.Bounds() == other.bounder_.Bounds();` ?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.ReadGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
Review Comment:
`null_count` is useless here.
##########
cpp/src/parquet/geospatial_statistics_test.cc:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_EQ(stats.get_geometry_types().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_TRUE(stats.is_empty());
+ EXPECT_FALSE(stats.has_z());
+ EXPECT_FALSE(stats.has_m());
+ EXPECT_EQ(stats.get_xmax() - stats.get_xmin(), -kInf);
+ EXPECT_EQ(stats.get_ymax() - stats.get_ymin(), -kInf);
+ EXPECT_EQ(stats.get_zmax() - stats.get_zmin(), -kInf);
+ EXPECT_EQ(stats.get_mmax() - stats.get_mmin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [inf, -inf]\n y: [inf, -inf]\n geometry_types:\n");
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeoStatistics());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeoStatistics, TestUpdateByteArray) {
+ GeoStatistics stats;
+
+ // Make items with all dimensions to ensure all dimensions are updated
+ // and returned properly (POINT XYZM has an integer code of 3001)
+ std::string xyzm0 = test::MakeWKBPoint({10, 11, 12, 13}, true, true);
+ ByteArray item0{xyzm0};
+
+ stats.Update(&item0, /*num_values=*/1, /*null_count=*/0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_upper_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_geometry_types(), ::testing::ElementsAre(3001));
+
+ std::string xyzm1 = test::MakeWKBPoint({20, 21, 22, 23}, true, true);
+ ByteArray item1{xyzm1};
+
+ stats.Update(&item1, /*num_values=*/1, /*null_count=*/0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_upper_bound(), ::testing::ElementsAre(20, 21, 22, 23));
+ EXPECT_THAT(stats.get_geometry_types(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [10, 20]\n y: [11, 21]\n z: [12, 22]\n m: "
+ "[13, 23]\n geometry_types: 3001\n");
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ std::string xyzm2 = test::MakeWKBPoint({-30, -31, -32, -33}, true, true);
+ ByteArray item2{xyzm2};
+
+ // A non-null value that shouldn't be skipped
+ std::string xyzm3 = test::MakeWKBPoint({30, 31, 32, 33}, true, true);
+ ByteArray item3{xyzm3};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeoStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats_spaced.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats_spaced.get_upper_bound(), ::testing::ElementsAre(30, 31, 32, 33));
+ EXPECT_THAT(stats_spaced.get_geometry_types(), ::testing::ElementsAre(3001));
Review Comment:
It would be good to verify `has_z()` and `has_m()`. BTW, is it possible to add cases for `xyz` and `xym`?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
Review Comment:
```suggestion
"Wraparound X or Y is not supported by GeoStatistics::Merge()");
```
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1152,13 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// \brief Set the GeoCrsContext with which coordinate reference systems should be
+ /// interpreted for GEOMETRY and GEOGRAPHY types
+ Builder* set_geospatial_crs_context(std::shared_ptr<GeoCrsContext> geo_crs_context) {
+ geo_crs_context_ = geo_crs_context;
Review Comment:
```suggestion
geo_crs_context_ = std::move(geo_crs_context);
```
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeoStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double get_xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no X values were
+ /// encountered, subject to "wrap around" bounds (see GetXMin()).
+ double get_xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no Y values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where ymin > ymax. In this
Review Comment:
Do we support wraparound for Y? Isn't Y is enforced to be latitude?
##########
cpp/src/parquet/metadata.cc:
##########
@@ -309,6 +321,14 @@ class ColumnChunkMetaData::ColumnChunkMetaDataImpl {
descr_->sort_order());
}
+ inline bool is_geo_stats_set() const {
+ if (possible_geo_stats_ == nullptr &&
+ column_metadata_->__isset.geospatial_statistics) {
+ possible_geo_stats_ = MakeColumnGeometryStats(*column_metadata_, descr_);
Review Comment:
I would prefer calling this proactively in the constructor and do not initialize it lazily. This is not thread-safe which makes it harder to cache and share the FileMetaData.
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -259,6 +262,12 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # TODO(paleolimbot): Make sure this is OK or remove if not!
+ if(ARROW_JSON)
Review Comment:
Is it better to add a new option like `PARQUET_SUPPORT_GEOARROW` similar to `PARQUET_REQUIRE_ENCRYPTION` to make it explicit?
##########
cpp/src/parquet/properties.h:
##########
@@ -69,6 +69,45 @@ constexpr int32_t kDefaultThriftContainerSizeLimit = 1000 * 1000;
// PARQUET-978: Minimize footer reads by reading 64 KB from the end of the file
constexpr int64_t kDefaultFooterReadSize = 64 * 1024;
+/// \brief Convert Coordinate Reference System values for GEOMETRY and GEOGRAPHY logical
+/// types
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoCrsContext {
+ public:
+ virtual ~GeoCrsContext() = default;
+
+ /// \brief Remove any previously saved PROJJSON CRS fields
+ virtual void Clear() = 0;
+
+ /// \brief Given a coordinate reference system value and encoding from GeoArrow
+ /// extension metadata, return the value that should be placed in the
+ /// LogicalType::Geography|Geometry(crs=) field
+ ///
+ /// For PROJJSON Crses (the most common way coordinate reference systems arrive
+ /// in Arrow), the Parquet specification forces us to write them to the file
+ /// metadata. The default GeoCrsContext will record such values and return the required
+ /// string that can be placed into the 'crs' of the Geometry or Geography logical
+ /// type.
+ virtual std::string GetParquetCrs(std::string crs_value,
Review Comment:
nit: add documentation for accepted values of `crs_encoding`?
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,241 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+namespace {
+
+::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMin);
+ auto max_geometry_type_value = static_cast<uint32_t>(GeometryType::kWKBValueMax);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMin);
+ auto max_dimension_value = static_cast<uint32_t>(Dimensions::kWKBValueMax);
+
+ if (geometry_type_component < min_geometry_type_value ||
+ geometry_type_component > max_geometry_type_value ||
+ dimensions_component < min_dimension_value ||
+ dimensions_component > max_dimension_value) {
+ return ::arrow::Status::SerializationError("Invalid WKB geometry type: ",
+ wkb_geometry_type);
+ }
+
+ GeometryTypeAndDimensions out{static_cast<GeometryType>(geometry_type_component),
+ static_cast<Dimensions>(dimensions_component)};
+ return out;
+}
+
+} // namespace
+
+std::vector<int32_t> WKBGeometryBounder::GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometry(std::string_view bytes_wkb) {
+ WKBBuffer src{reinterpret_cast<const uint8_t*>(bytes_wkb.data()),
+ static_cast<int64_t>(bytes_wkb.size())};
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(&src, /*record_wkb_type=*/true));
+ if (src.size() != 0) {
+ return ::arrow::Status::SerializationError(
+ "Exepcted zero bytes after consuming WKB but got ", src.size());
+ }
+
+ return ::arrow::Status::OK();
+}
+
+::arrow::Status WKBGeometryBounder::ReadGeometryInternal(WKBBuffer* src,
+ bool record_wkb_type) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
Review Comment:
Can we use structured binding to make it more explicit?
##########
cpp/src/parquet/geospatial_util_internal_json.h:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/properties.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromGeoArrowJSON(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties);
+
+/// \brief Compute a suitable DataType into which a GEOMETRY or GEOGRAPHY type should be
+/// read
+///
+/// The result of this function depends on whether or not "geoarrow.wkb" has been
+/// registered: if it has, the result will be the registered ExtensionType; if it has not,
+/// the result will be binary().
+::arrow::Result<std::shared_ptr<::arrow::DataType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata);
+
+/// \brief The default GeoCrsContext, which writes any PROJJSON coordinate reference
+/// system values it encounters to the file metadata
+class PARQUET_EXPORT FileGeoCrsContext : public GeoCrsContext {
Review Comment:
Will we add encoding other than PROJJSON? I'm thinking about the naming which depends on whether it is PROJJSON-specific or GeoArrow-specific.
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -428,13 +447,19 @@ Status FieldToNode(const std::string& name, const std::shared_ptr<Field>& field,
}
case ArrowTypeId::EXTENSION: {
auto ext_type = std::static_pointer_cast<::arrow::ExtensionType>(field->type());
- // Built-in JSON extension is handled differently.
+ // Built-in JSON extension and GeoArrow are handled differently.
Review Comment:
I was thinking if we can add an extension type registry which allows users to register an extension name and the function to create the extension type and/or parquet type. This is similar to what I've added to avro recently: https://github.com/apache/avro/pull/3326/files#diff-00521bd02440832a207504b570a0fc02ccbe0928c6710e419503d48284aeee90R95-R114
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -243,6 +245,23 @@ static Status GetTimestampMetadata(const ::arrow::TimestampType& type,
return Status::OK();
}
+Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromArrow(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties) {
+ // Without a JSON parser, we can still handle a few trivial cases
Review Comment:
Are these the most common (or default) cases?
##########
cpp/src/parquet/geospatial_statistics.cc:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_statistics.h"
+#include <cmath>
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
+ return false;
+ }
+
+ if (bounder_.Bounds() != other.bounder_.Bounds()) {
+ return false;
+ }
+
+ return true;
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || is_wraparound_y() || other.is_wraparound_x() ||
+ other.is_wraparound_y()) {
+ throw ParquetException(
+ "Wraparound X or Y is not suppored by GeoStatistics::Merge()");
Review Comment:
BTW, do we want to fail the writer or silently disable bbox?
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -243,6 +245,23 @@ static Status GetTimestampMetadata(const ::arrow::TimestampType& type,
return Status::OK();
}
+Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromArrow(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties) {
+ // Without a JSON parser, we can still handle a few trivial cases
Review Comment:
Why not putting all these logics into `GeospatialLogicalTypeFromGeoArrowJSON` so that they are not scattered in two separate places?
##########
cpp/src/parquet/properties.h:
##########
@@ -1112,6 +1152,13 @@ class PARQUET_EXPORT ArrowWriterProperties {
return this;
}
+ /// \brief Set the GeoCrsContext with which coordinate reference systems should be
+ /// interpreted for GEOMETRY and GEOGRAPHY types
+ Builder* set_geospatial_crs_context(std::shared_ptr<GeoCrsContext> geo_crs_context) {
+ geo_crs_context_ = geo_crs_context;
Review Comment:
Should we make it thread-safe? There are cases where a single ReaderProperties is shared by multiple readers (e.g. in the dataset scanner) and have concurrent access to it.
##########
cpp/src/parquet/arrow/writer.cc:
##########
@@ -546,6 +554,10 @@ Status GetSchemaMetadata(const ::arrow::Schema& schema, ::arrow::MemoryPool* poo
std::string schema_as_string = serialized->ToString();
std::string schema_base64 = ::arrow::util::base64_encode(schema_as_string);
result->Append(kArrowSchemaKey, std::move(schema_base64));
+
+ // Add any required fields collected by the GeoCrsContext
+ properties.geo_crs_context()->AddProjjsonCrsFieldsToFileMetadata(result.get());
Review Comment:
It is weird to mutate the member variable of a const reference. We can either add GeoCrsContext as a separate parameter or change ArrowWriterProperties to be non-const.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007965479
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -428,13 +447,19 @@ Status FieldToNode(const std::string& name, const std::shared_ptr<Field>& field,
}
case ArrowTypeId::EXTENSION: {
auto ext_type = std::static_pointer_cast<::arrow::ExtensionType>(field->type());
- // Built-in JSON extension is handled differently.
+ // Built-in JSON extension and GeoArrow are handled differently.
Review Comment:
Yes, particularly with https://github.com/apache/arrow/pull/45866 it's apparent that there is some common code among extension types that could perhaps be separated (and with variant we'll be up to four extension types to/from five parquet types). I could try that here but I wonder if it would easier to review in a dedicated change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008354319
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1227,10 +1239,14 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
// Will be null if not using dictionary, but that's ok
current_dict_encoder_ = dynamic_cast<DictEncoder<DType>*>(current_encoder_.get());
- if (properties->statistics_enabled(descr_->path()) &&
- (SortOrder::UNKNOWN != descr_->sort_order())) {
- page_statistics_ = MakeStatistics<DType>(descr_, allocator_);
- chunk_statistics_ = MakeStatistics<DType>(descr_, allocator_);
+ if (properties->statistics_enabled(descr_->path())) {
+ if (SortOrder::UNKNOWN != descr_->sort_order()) {
Review Comment:
I added a comment here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059542185
##########
cpp/src/parquet/geospatial/statistics.cc:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial/statistics.h"
+
+#include <cmath>
+#include <memory>
+#include <optional>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "parquet/exception.h"
+#include "parquet/geospatial/util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet::geospatial {
+
+class GeoStatisticsImpl {
+ public:
+ bool Equals(const GeoStatisticsImpl& other) const {
+ return is_valid_ == other.is_valid_ &&
+ bounder_.GeometryTypes() == other.bounder_.GeometryTypes() &&
+ bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeoStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x() || other.is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not supported by GeoStatistics::Merge()");
+ }
+
+ bounder_.MergeBox(other.bounder_.Bounds());
+ std::vector<int32_t> other_geometry_types = other.bounder_.GeometryTypes();
+ bounder_.MergeGeometryTypes(other_geometry_types);
+ }
+
+ void Update(const ByteArray* values, int64_t num_values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ try {
+ bounder_.MergeGeometry({reinterpret_cast<const char*>(item.ptr), item.len});
+ } catch (ParquetException&) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ PARQUET_CATCH_NOT_OK(bounder_.MergeGeometry(
+ {reinterpret_cast<const char*>(item.ptr), item.len}));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ if (is_wraparound_x()) {
+ throw ParquetException("Wraparound X is not suppored by GeoStatistics::Update()");
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException("Unsupported Array type in GeoStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeoStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geospatial::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geospatial::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeoStatistics out;
+
+ if (has_geometry_types_) {
+ out.geospatial_types = bounder_.GeometryTypes();
+ }
+
+ bool write_x = !bound_empty(0) && bound_valid(0);
+ bool write_y = !bound_empty(1) && bound_valid(1);
+ bool write_z = write_x && write_y && !bound_empty(2) && bound_valid(2);
+ bool write_m = write_x && write_y && !bound_empty(3) && bound_valid(3);
+
+ if (write_x && write_y) {
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.xy_bounds_present = true;
+ }
+
+ if (write_z) {
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.z_bounds_present = true;
+ }
+
+ if (write_m) {
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+ out.m_bounds_present = true;
+ }
+
+ return out;
+ }
+
+ void Update(const EncodedGeoStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // We can create GeoStatistics from a wraparound bounding box, but we can't
+ // update an existing one because the merge logic is not yet implemented.
Review Comment:
It is probably not hard but I'd rather not implement it in this PR since there's quite a bit here. This particular line it turns out was never called on a non-empty value, so I made that explicit (by renaming this to Decode() and moving the Reset() call and removing this check). The other place that threw for a wraparound I changed to mark the X values as invalid instead of throw. There aren't any writers writing these boxes yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059543564
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,136 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types (experimental)"""
+
+ def __init__(self):
+ raise TypeError(f"Do not call {self.__class__.__name__}'s constructor directly")
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ cdef optional[vector[int32_t]] maybe_geometry_types = \
+ self.statistics.get().geometry_types()
+ if not maybe_geometry_types.has_value():
+ return None
+
+ return list(maybe_geometry_types.value())
+
+ @property
+ def lower_bound(self):
+ return [self.statistics.get().lower_bound()[i] for i in range(4)]
+
+ @property
+ def upper_bound(self):
+ return [self.statistics.get().upper_bound()[i] for i in range(4)]
+
+ @property
+ def dimension_empty(self):
+ return [self.statistics.get().dimension_empty()[i] for i in range(4)]
+
+ @property
+ def dimension_valid(self):
+ return [self.statistics.get().dimension_valid()[i] for i in range(4)]
+
+ @property
+ def has_x(self):
+ return self.dimension_valid[0] and not self.dimension_empty[0]
+
+ @property
+ def has_y(self):
+ return self.dimension_valid[1] and not self.dimension_empty[1]
+
+ @property
+ def has_z(self):
+ return self.dimension_valid[2] and not self.dimension_empty[2]
+
+ @property
+ def has_m(self):
+ return self.dimension_valid[3] and not self.dimension_empty[3]
+
+ @property
+ def xmin(self):
+ if self.has_x:
+ return self.lower_bound[0]
+ else:
+ return None
Review Comment:
I also updated `has_x` to do something similar!
##########
python/pyarrow/_parquet.pyx:
##########
@@ -450,6 +590,24 @@ cdef class ColumnChunkMetaData(_Weakrefable):
statistics.init(self.metadata.statistics(), self)
return statistics
+ @property
+ def is_geo_stats_set(self):
+ """Whether or not geometry statistics are present in metadata (bool)."""
+ return self.metadata.is_geo_stats_set()
+
+ @property
+ def geo_statistics(self):
+ """Statistics for column chunk (:class:`GeoStatistics`)."""
+ if not self.metadata.is_geo_stats_set():
+ return None
+
+ if not self.metadata.geo_statistics().get().is_valid():
+ return None
+
+ cdef GeoStatistics geo_statistics = GeoStatistics.__new__(GeoStatistics)
+ geo_statistics.init(self.metadata.geo_statistics(), self)
+ return geo_statistics
Review Comment:
Works great! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057948393
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
It seems that https://github.com/apache/parquet-cpp/pull/279 introduced it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057975063
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
> Do we need to modify the CMake config to enable `ARROW_JSON` when `ARROW_PARQUET` is ON?
`ARROW_JSON` will build the Arrow JSONL reader, which should not be required for Parquet. We just need RapidJSON here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059088194
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1826,6 +1870,12 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
page_statistics_->IncrementNullCount(num_chunk_levels - non_null_count);
page_statistics_->IncrementNumValues(non_null_count);
page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false);
+
+ if (chunk_geospatial_statistics_ != nullptr) {
+ throw ParquetException(
+ "Writing dictionary-encoded GEOMETRY or GEOGRAPHY with statistics is not "
+ "supported");
+ }
Review Comment:
I'm not sure if this will ever come up on purpose...it's definitely not a typical use case to store a lot of repeated geometries. If this happens by accident/automatically for a reason I haven't learned about yet I'm happy to circle back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2826346276
I think this is ready for another round when time allows! A few notes:
- @jorisvandenbossche mentioned in a recent GeoParquet sync that it might be nice if the GEOMETRY type was written even if the type was not registered (i.e., only field metadata were present). I'd like to avoid that in this PR just because it's also a change with respect to the other types we currently handle.
- The bounding boxes I believe are synced with our discussion at https://github.com/apache/parquet-format/pull/494 . I opened a follow-up to ensure we can get a null count for geospatial types and I'm happy to give that a go after this merges.
- The example files are written using the latest version of this, and include some Python tests. I'm happy to move the Python tests here after the example files are in parquet-testing ( https://github.com/apache/parquet-testing/pull/70 ).
I know this has gotten quite large but if possible I hope this can merge relatively early in the 21.0.0 release cycle so that we (and others) can experiment with the nightly Python builds and identify issues arising from some target use cases. Let me know what any next steps are on my end!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016102865
##########
cpp/src/parquet/geospatial_statistics.h:
##########
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_x() const { return !std::isinf(xmin - xmax); }
+ bool has_y() const { return !std::isinf(ymin - ymax); }
+ bool has_z() const { return !std::isinf(zmin - zmax); }
+ bool has_m() const { return !std::isinf(mmin - mmax); }
+
+ bool is_set() const {
+ return !geospatial_types.empty() || has_x() || has_y() || has_z() || has_m();
+ }
+};
+
+class GeoStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+/// or representing them when reading a file
+///
+/// Note that NaN values that were encountered within coordinates are omitted; however,
+/// NaN values that were obtained via decoding encoded statistics are propagated. This
+/// behaviour ensures C++ clients that are inspecting statistics via the column metadata
+/// can detect the case where a writer generated NaNs (even though this implementation
+/// does not generate them).
+///
+/// The handling of NaN values in coordinates is not well-defined among bounding
+/// implementations except for the WKB convention for POINT EMPTY, which is consistently
+/// represented as a point whose ordinates are all NaN. Any other geometry that contains
+/// NaNs cannot expect defined behaviour here or elsewhere; however, a row group that
+/// contains both NaN-containing and normal (completely finite) geometries should not be
+/// excluded from predicate pushdown.
+///
+/// EXPERIMENTAL
+class PARQUET_EXPORT GeoStatistics {
+ public:
+ GeoStatistics();
+ explicit GeoStatistics(const EncodedGeoStatistics& encoded);
+
+ ~GeoStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeoStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeoStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeoStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ /// \brief Reset existing statistics and populate them from previously-encoded ones
+ void Decode(const EncodedGeoStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no non-NaN X
+ /// values were encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other writers.
+ double xmin() const;
+
+ /// \brief The maximum encountered value in the X dimension, or -Inf if no non-NaN X
+ /// values were encountered, subject to "wrap around" bounds (see xmin()).
+ double xmax() const;
+
+ /// \brief The minimum encountered value in the Y dimension, or Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymin() const;
+
+ /// \brief The maximum encountered value in the Y dimension, or -Inf if no non-NaN Y
+ /// values were encountered. Wrap around bounds are not permitted in the Y dimension.
+ double ymax() const;
+
+ /// \brief The minimum encountered value in the Z dimension, or Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmin() const;
+
+ /// \brief The maximum encountered value in the Z dimension, or -Inf if no non-NaN Z
+ /// values were encountered. Wrap around bounds are not permitted in the Z dimension.
+ double zmax() const;
+
+ /// \brief The minimum encountered value in the M dimension, or Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmin() const;
+
+ /// \brief The maximum encountered value in the M dimension, or -Inf if no non-NaN M
+ /// values were encountered. Wrap around bounds are not permitted in the M dimension.
+ double mmax() const;
+
+ /// \brief All minimum values in XYZM order
+ std::array<double, 4> lower_bound() const;
+
+ /// \brief All maximum values in XYZM order
+ std::array<double, 4> upper_bound() const;
+
+ /// \brief Returns true if zero finite coordinates or geometry types were encountered
+ ///
+ /// This will occur, for example, if all encountered values were null. If all
+ /// encountered values were EMPTY (e.g., all values were POINT EMPTY, LINESTRING EMPTY,
+ /// etc.), is_empty() will still return false because the geometry types list will
+ /// contain values.
+ bool is_empty() const;
+
+ /// \brief Returns true if any non-NaN X values were encountered or false otherwise
+ bool has_x() const;
Review Comment:
Do we also want a convenience method to return all 4 predicates? Something like:
```c++
/// \brief Return whether each of X, Y, Z, M coordinate is present
std::array<bool, 4> has_coordinates() const;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016824600
##########
cpp/src/parquet/geospatial_util_json_internal.cc:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_json_internal.h"
+
+#include <string>
+
+#include "arrow/config.h"
+#include "arrow/extension_type.h"
+#include "arrow/result.h"
+#include "arrow/util/string.h"
+
+#include "parquet/exception.h"
+#include "parquet/types.h"
+
+#ifdef ARROW_JSON
Review Comment:
The Arrow-vendored RapidJSON uses the `arrow::rapidjson` namespace, so that sounds quite unexpected in any case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016156526
##########
python/pyarrow/_parquet.pyx:
##########
@@ -319,6 +319,84 @@ cdef _box_flba(ParquetFLBA val, uint32_t len):
return cp.PyBytes_FromStringAndSize(<char*> val.ptr, <Py_ssize_t> len)
+cdef class GeoStatistics(_Weakrefable):
+ """Statistics for columns with geospatial data types"""
+
+ def __cinit__(self):
+ pass
+
+ def __repr__(self):
+ return f"""{object.__repr__(self)}
+ geospatial_types: {self.geospatial_types}
+ xmin: {self.xmin}, xmax: {self.xmax}
+ ymin: {self.ymin}, ymax: {self.ymax}
+ zmin: {self.zmin}, zmax: {self.zmax}
+ mmin: {self.mmin}, mmax: {self.mmax}"""
+
+ def to_dict(self):
+ out = {
+ "geospatial_types": self.geospatial_types,
+ "xmin": self.xmin,
+ "xmax": self.xmax,
+ "ymin": self.ymin,
+ "ymax": self.ymax,
+ "zmin": self.zmin,
+ "zmax": self.zmax,
+ "mmin": self.mmin,
+ "mmax": self.mmax
+ }
+
+ return out
+
+ @property
+ def geospatial_types(self):
+ return list(self.statistics.geospatial_types)
+
+ @property
+ def xmin(self):
+ return self.statistics.xmin
Review Comment:
I don't understand, is this assuming that `has_x` (and `has_y`) always returns true on the C++ side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2016122743
##########
cpp/src/parquet/geospatial_util_internal.cc:
##########
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geospatial_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(nullptr), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
Review Comment:
I don't think you want `ARROW_PREDICT_FALSE` here, because it can be very likely if you read a data file with non-matching endianness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1962884901
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
Review Comment:
Should we skip the following if is_valid_ is false?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
Review Comment:
```suggestion
std::vector<int32_t> GeospatialTypes() const {
```
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ is_valid_ = is_valid_ && other.is_valid_;
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ ::arrow::Status status = bounder_.ReadGeometry(item.ptr, item.len);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(item.ptr, item.len));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ // Note that ::arrow::Type::EXTENSION seems to be handled before this is called
+ switch (values.type_id()) {
+ case ::arrow::Type::BINARY:
+ UpdateArrayImpl<::arrow::BinaryArray>(values);
+ break;
+ case ::arrow::Type::LARGE_BINARY:
+ UpdateArrayImpl<::arrow::LargeBinaryArray>(values);
+ break;
+ // This does not currently handle run-end encoded, dictionary encodings, or views
+ default:
+ throw ParquetException(
+ "Unsupported Array type in GeospatialStatistics::Update(Array): ",
+ values.type()->ToString());
+ }
+ }
+
+ void Reset() {
+ bounder_.Reset();
+ is_valid_ = true;
+ }
+
+ EncodedGeospatialStatistics Encode() const {
+ if (!is_valid_) {
+ return {};
+ }
+
+ const geometry::BoundingBox::XYZM& mins = bounder_.Bounds().min;
+ const geometry::BoundingBox::XYZM& maxes = bounder_.Bounds().max;
+
+ EncodedGeospatialStatistics out;
+ out.geospatial_types = bounder_.GeometryTypes();
+
+ out.xmin = mins[0];
+ out.xmax = maxes[0];
+ out.ymin = mins[1];
+ out.ymax = maxes[1];
+ out.zmin = mins[2];
+ out.zmax = maxes[2];
+ out.mmin = mins[3];
+ out.mmax = maxes[3];
+
+ return out;
+ }
+
+ void Update(const EncodedGeospatialStatistics& encoded) {
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::BoundingBox box;
+ box.min[0] = encoded.xmin;
+ box.max[0] = encoded.xmax;
+ box.min[1] = encoded.ymin;
+ box.max[1] = encoded.ymax;
+
+ if (encoded.has_z()) {
+ box.min[2] = encoded.zmin;
+ box.max[2] = encoded.zmax;
+ }
+
+ if (encoded.has_m()) {
+ box.min[3] = encoded.mmin;
+ box.max[3] = encoded.mmax;
+ }
+
+ bounder_.ReadBox(box);
+ bounder_.ReadGeometryTypes(encoded.geospatial_types);
+ }
+
+ bool is_valid() const { return is_valid_; }
+
+ const std::array<double, 4>& GetMinBounds() const { return bounder_.Bounds().min; }
+
+ const std::array<double, 4>& GetMaxBounds() { return bounder_.Bounds().max; }
+
+ std::vector<int32_t> GetGeometryTypes() const { return bounder_.GeometryTypes(); }
+
+ private:
+ geometry::WKBGeometryBounder bounder_;
+ bool is_valid_ = true;
+
+ template <typename ArrayType>
+ void UpdateArrayImpl(const ::arrow::Array& values) {
+ const auto& binary_array = static_cast<const ArrayType&>(values);
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
+ if (!binary_array.IsNull(i)) {
+ std::string_view byte_array = binary_array.GetView(i);
+ ::arrow::Status status = bounder_.ReadGeometry(
+ reinterpret_cast<const uint8_t*>(byte_array.data()), byte_array.length());
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+ }
+};
+
+GeospatialStatistics::GeospatialStatistics()
+ : impl_(std::make_unique<GeospatialStatisticsImpl>()) {}
+
+GeospatialStatistics::GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl)
+ : impl_(std::move(impl)) {}
+
+GeospatialStatistics::GeospatialStatistics(const EncodedGeospatialStatistics& encoded)
+ : GeospatialStatistics() {
+ Decode(encoded);
+}
+
+GeospatialStatistics::GeospatialStatistics(GeospatialStatistics&&) = default;
+
+GeospatialStatistics::~GeospatialStatistics() = default;
+
+bool GeospatialStatistics::Equals(const GeospatialStatistics& other) const {
+ return impl_->Equals(*other.impl_);
+}
+
+void GeospatialStatistics::Merge(const GeospatialStatistics& other) {
+ impl_->Merge(*other.impl_);
+}
+
+void GeospatialStatistics::Update(const ByteArray* values, int64_t num_values,
+ int64_t null_count) {
+ impl_->Update(values, num_values, null_count);
+}
+
+void GeospatialStatistics::UpdateSpaced(const ByteArray* values,
+ const uint8_t* valid_bits,
+ int64_t valid_bits_offset,
+ int64_t num_spaced_values, int64_t num_values,
+ int64_t null_count) {
+ impl_->UpdateSpaced(values, valid_bits, valid_bits_offset, num_spaced_values,
+ num_values, null_count);
+}
+
+void GeospatialStatistics::Update(const ::arrow::Array& values) { impl_->Update(values); }
+
+void GeospatialStatistics::Reset() { impl_->Reset(); }
+
+bool GeospatialStatistics::is_valid() const { return impl_->is_valid(); }
+
+EncodedGeospatialStatistics GeospatialStatistics::Encode() const {
+ return impl_->Encode();
+}
+
+void GeospatialStatistics::Decode(const EncodedGeospatialStatistics& encoded) {
+ impl_->Update(encoded);
Review Comment:
Should we reset it before calling update?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
Review Comment:
Is it able to deal with `geography` type or a separate bounder is required?
##########
cpp/src/parquet/geometry_util_internal.cc:
##########
@@ -0,0 +1,234 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
Review Comment:
Should we use `span` instead?
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
Review Comment:
```suggestion
enum class GeospatialType {
```
Though I think `GeometryType` sounds better :)
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
Review Comment:
This only applies to `geometry` type, right? For `geography` type, we need to deal with wraparound and edges?
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
Review Comment:
```suggestion
if (bounder_.GeometryTypes() != other.bounder_.GeometryTypes()) {
return false;
}
```
I think this should work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2671868876
> I believe that I have reviewed all files except testing code. Thank you for the great work!
Thank you! I'll work through this in detail today 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2024945324
##########
cpp/src/parquet/geospatial_util_internal.h:
##########
@@ -0,0 +1,226 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cmath>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/util/logging.h"
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+///
+/// These values correspond to the 0, 1000, 2000, 3000 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type // 1000).
+enum class Dimensions {
+ kXY = 0,
+ kXYZ = 1,
+ kXYM = 2,
+ kXYZM = 3,
+ kValueMin = 0,
+ kValueMax = 3
+};
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+///
+/// These values correspond to the 1, 2, ..., 7 component of the WKB integer
+/// geometry type (i.e., the value of geometry_type % 1000).
+enum class GeometryType {
+ kPoint = 1,
+ kLinestring = 2,
+ kPolygon = 3,
+ kMultiPoint = 4,
+ kMultiLinestring = 5,
+ kMultiPolygon = 6,
+ kGeometryCollection = 7,
+ kValueMin = 1,
+ kValueMax = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+///
+/// The Parquet specification also supports wraparound bounding boxes in the X
+/// dimension; however, this structure assumes min < max always as it is used for
+/// the purposes of accumulating this type of bounds.
+///
+/// This class will ignore any NaN values it visits via UpdateXY[Z[M]](). This is
+/// consistent with GEOS and ensures that ranges are accumulated in the same way that
+/// other statistics in Parquet are accumulated (e.g., statistics of a double column will
+/// return a min/max of all non-NaN values). In WKB specifically, POINT EMPTY is
+/// represented by convention as all ordinate values filled with NaN, so this behaviour
+/// allows for no special-casing of POINT EMPTY in the WKB reader.
+///
+/// This class will propagate any NaN values (per dimension) that were explicitly
+/// specified via setting mins/maxes directly or by merging another BoundingBox that
+/// contained NaN values. This definition ensures that NaN bounds obtained via
+/// EncodedGeoStatistics (which may have been written by some other writer that generated
+/// NaNs, either on purpose or by accident) are not silently overwritten.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 2);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 3);
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(::arrow::util::span<const double> coord) {
+ DCHECK_EQ(coord.size(), 4);
+ UpdateInternal(coord);
+ }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ if (std::isnan(min[i]) || std::isnan(max[i]) || std::isnan(other.min[i]) ||
+ std::isnan(other.max[i])) {
+ min[i] = std::numeric_limits<double>::quiet_NaN();
+ max[i] = std::numeric_limits<double>::quiet_NaN();
+ } else {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox" << std::endl;
+ ss << " x: [" << min[0] << ", " << max[0] << "]" << std::endl;
+ ss << " y: [" << min[1] << ", " << max[1] << "]" << std::endl;
+ ss << " z: [" << min[2] << ", " << max[2] << "]" << std::endl;
+ ss << " m: [" << min[3] << ", " << max[3] << "]" << std::endl;
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+inline bool operator!=(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return !(lhs == rhs);
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+///
+/// Note that this class is NOT appropriate for bounding a GEOGRAPHY,
+/// whose bounds are not a function purely of the vertices. Geography bounding
+/// is not yet implemented.
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ::arrow::Status MergeGeometry(std::string_view bytes_wkb);
Review Comment:
I think @emkornfield's point is that these utility functions might be moved out of parquet in the future. I'm inclined to throw ParquetException for now because they are actually internal functions and can be easily rewritten to use Status once we need to move them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007942940
##########
cpp/src/parquet/geospatial_util_internal_json.h:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+
+#include "arrow/util/key_value_metadata.h"
+
+#include "parquet/properties.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Compute a Parquet Logical type (Geometry(...) or Geography(...)) from
+/// serialized GeoArrow metadata
+///
+/// Returns the appropriate LogicalType, SerializationError if the metadata was invalid,
+/// or NotImplemented if Parquet was not built with ARROW_JSON.
+::arrow::Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromGeoArrowJSON(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties);
+
+/// \brief Compute a suitable DataType into which a GEOMETRY or GEOGRAPHY type should be
+/// read
+///
+/// The result of this function depends on whether or not "geoarrow.wkb" has been
+/// registered: if it has, the result will be the registered ExtensionType; if it has not,
+/// the result will be binary().
+::arrow::Result<std::shared_ptr<::arrow::DataType>> MakeGeoArrowGeometryType(
+ const LogicalType& logical_type,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata);
+
+/// \brief The default GeoCrsContext, which writes any PROJJSON coordinate reference
+/// system values it encounters to the file metadata
+class PARQUET_EXPORT FileGeoCrsContext : public GeoCrsContext {
+ public:
+ FileGeoCrsContext() { Clear(); }
+
+ void Clear() override {
+ projjson_crs_fields_ = ::arrow::KeyValueMetadata::Make({}, {});
+ }
+
+ std::string GetParquetCrs(std::string crs_value,
+ const std::string& crs_encoding) override;
+
+ bool HasProjjsonCrsFields() override { return projjson_crs_fields_->size() > 0; }
+
+ void AddProjjsonCrsFieldsToFileMetadata(::arrow::KeyValueMetadata* metadata) override;
+
+ private:
+ std::shared_ptr<::arrow::KeyValueMetadata> projjson_crs_fields_;
Review Comment:
I'll write a more detailed note in a top-level comment below, but I think there are a sufficient number of implementation considerations with the `GeoCrsContext` that it should be dealt with separately (I removed it from this PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007939590
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -259,6 +262,12 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # TODO(paleolimbot): Make sure this is OK or remove if not!
+ if(ARROW_JSON)
Review Comment:
I am happy to do either here, although I did notice that there are >60 references to `PARQUET_REQUIRE_ENCRYPTION` in our codebase and so I'd like to be very sure that this is the desired direction before replicating those references to ensure geoarrow support. (`ARROW_JSON` is the mechananism that the canonical extension types that require JSON parsing use to determine whether they are or are not supported in a given Arrow C++ build).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2007958395
##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -243,6 +245,23 @@ static Status GetTimestampMetadata(const ::arrow::TimestampType& type,
return Status::OK();
}
+Result<std::shared_ptr<const LogicalType>> GeospatialLogicalTypeFromArrow(
+ const std::string& serialized_data, const ArrowWriterProperties& arrow_properties) {
+ // Without a JSON parser, we can still handle a few trivial cases
Review Comment:
I moved all of this logic to `geospatial_util_internal_json.cc`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2008622743
##########
cpp/src/parquet/geospatial_statistics_test.cc:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+
+#include "arrow/array/builder_binary.h"
+#include "arrow/compute/api.h"
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geospatial_statistics.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeoStatistics, TestDefaults) {
+ GeoStatistics stats;
+ EXPECT_EQ(stats.get_geometry_types().size(), 0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_TRUE(stats.is_empty());
+ EXPECT_FALSE(stats.has_z());
+ EXPECT_FALSE(stats.has_m());
+ EXPECT_EQ(stats.get_xmax() - stats.get_xmin(), -kInf);
+ EXPECT_EQ(stats.get_ymax() - stats.get_ymin(), -kInf);
+ EXPECT_EQ(stats.get_zmax() - stats.get_zmin(), -kInf);
+ EXPECT_EQ(stats.get_mmax() - stats.get_mmin(), -kInf);
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [inf, -inf]\n y: [inf, -inf]\n geometry_types:\n");
+
+ auto encoded = stats.Encode();
+ EXPECT_FALSE(encoded.is_set());
+ EXPECT_FALSE(encoded.has_z());
+ EXPECT_FALSE(encoded.has_m());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+
+ stats.Merge(GeoStatistics());
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+}
+
+TEST(TestGeoStatistics, TestUpdateByteArray) {
+ GeoStatistics stats;
+
+ // Make items with all dimensions to ensure all dimensions are updated
+ // and returned properly (POINT XYZM has an integer code of 3001)
+ std::string xyzm0 = test::MakeWKBPoint({10, 11, 12, 13}, true, true);
+ ByteArray item0{xyzm0};
+
+ stats.Update(&item0, /*num_values=*/1, /*null_count=*/0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_upper_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_geometry_types(), ::testing::ElementsAre(3001));
+
+ std::string xyzm1 = test::MakeWKBPoint({20, 21, 22, 23}, true, true);
+ ByteArray item1{xyzm1};
+
+ stats.Update(&item1, /*num_values=*/1, /*null_count=*/0);
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats.get_upper_bound(), ::testing::ElementsAre(20, 21, 22, 23));
+ EXPECT_THAT(stats.get_geometry_types(), ::testing::ElementsAre(3001));
+
+ // Check recreating the statistics with actual values
+ auto encoded = stats.Encode();
+ EXPECT_TRUE(GeoStatistics(encoded).Equals(stats));
+ EXPECT_EQ(stats.ToString(),
+ "GeoStatistics \n x: [10, 20]\n y: [11, 21]\n z: [12, 22]\n m: "
+ "[13, 23]\n geometry_types: 3001\n");
+
+ // Check resetting to the original state
+ stats.Reset();
+ EXPECT_TRUE(stats.Equals(GeoStatistics()));
+
+ // Check UpdateSpaced()
+
+ // A null value that should be skipped
+ std::string xyzm2 = test::MakeWKBPoint({-30, -31, -32, -33}, true, true);
+ ByteArray item2{xyzm2};
+
+ // A non-null value that shouldn't be skipped
+ std::string xyzm3 = test::MakeWKBPoint({30, 31, 32, 33}, true, true);
+ ByteArray item3{xyzm3};
+
+ ByteArray items[] = {item0, item1, item2, item3};
+ // Validity bitmap with an extra bit on the front to check non-zero bits offset
+ uint8_t validity = 0b00010111;
+ GeoStatistics stats_spaced;
+ stats_spaced.UpdateSpaced(items, &validity, 1, 4, 4, 1);
+
+ EXPECT_TRUE(stats.is_valid());
+ EXPECT_THAT(stats_spaced.get_lower_bound(), ::testing::ElementsAre(10, 11, 12, 13));
+ EXPECT_THAT(stats_spaced.get_upper_bound(), ::testing::ElementsAre(30, 31, 32, 33));
+ EXPECT_THAT(stats_spaced.get_geometry_types(), ::testing::ElementsAre(3001));
Review Comment:
Test added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Proof-of-concept Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1955569504
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
+
+ return bounder_.Bounds() == other.bounder_.Bounds();
+ }
+
+ void Merge(const GeospatialStatisticsImpl& other) {
+ if (!is_valid_ || !other.is_valid_) {
+ is_valid_ = false;
+ return;
+ }
+
+ bounder_.ReadBox(other.bounder_.Bounds());
+ bounder_.ReadGeometryTypes(other.bounder_.GeometryTypes());
+ }
+
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count) {
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < num_values; i++) {
+ const ByteArray& item = values[i];
+ buf.Init(item.ptr, item.len);
+ ::arrow::Status status = bounder_.ReadGeometry(&buf);
+ if (!status.ok()) {
+ is_valid_ = false;
+ return;
+ }
+ }
+ }
+
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count) {
+ DCHECK_GT(num_spaced_values, 0);
+
+ if (!is_valid_) {
+ return;
+ }
+
+ geometry::WKBBuffer buf;
+ ::arrow::Status status = ::arrow::internal::VisitSetBitRuns(
+ valid_bits, valid_bits_offset, num_spaced_values,
+ [&](int64_t position, int64_t length) {
+ for (int64_t i = 0; i < length; i++) {
+ ByteArray item = SafeLoad(values + i + position);
+ buf.Init(item.ptr, item.len);
+ ARROW_RETURN_NOT_OK(bounder_.ReadGeometry(&buf));
+ }
+
+ return ::arrow::Status::OK();
+ });
+
+ if (!status.ok()) {
+ is_valid_ = false;
+ }
+ }
+
+ void Update(const ::arrow::Array& values) {
+ if (!is_valid_) {
+ return;
+ }
+
+ const auto& binary_array = static_cast<const ::arrow::BinaryArray&>(values);
+ geometry::WKBBuffer buf;
+ for (int64_t i = 0; i < binary_array.length(); ++i) {
Review Comment:
probably doesn't make much different perf wise for this case but `VisitArraySpanInline` is another way of doing this. For now this simpler code is probably better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1958668224
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,336 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3 };
+
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7
+};
+
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
+
+ void Init(const uint8_t* data, int64_t size) {
+ data_ = data;
+ size_ = size;
+ }
+
+ ::arrow::Result<uint8_t> ReadUInt8() { return ReadChecked<uint8_t>(); }
+
+ ::arrow::Result<uint32_t> ReadUInt32(bool swap) {
+ ARROW_ASSIGN_OR_RAISE(auto value, ReadChecked<uint32_t>());
+ if (ARROW_PREDICT_FALSE(swap)) {
+ return value = ByteSwap(value);
+ } else {
+ return value;
+ }
+ }
+
+ template <typename Coord, typename Visit>
+ ::arrow::Status ReadDoubles(uint32_t n_coords, bool swap, Visit&& visit) {
+ size_t total_bytes = n_coords * sizeof(Coord);
+ if (size_ < total_bytes) {
+ return ::arrow::Status::SerializationError(
+ "Can't coordinate sequence of ", total_bytes, " bytes from WKBBuffer with ",
+ size_, " remaining");
+ }
+
+ if (ARROW_PREDICT_FALSE(swap)) {
+ Coord coord;
+ for (uint32_t i = 0; i < n_coords; i++) {
+ coord = ReadUnchecked<Coord>();
+ for (uint32_t j = 0; j < coord.size(); j++) {
+ coord[j] = ByteSwap(coord[j]);
+ }
+
+ visit(coord);
+ }
+ } else {
+ for (uint32_t i = 0; i < n_coords; i++) {
+ visit(ReadUnchecked<Coord>());
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ size_t size() { return size_; }
+
+ private:
+ const uint8_t* data_;
+ size_t size_;
+
+ template <typename T>
+ ::arrow::Result<T> ReadChecked() {
+ if (size_ < sizeof(T)) {
+ return ::arrow::Status::SerializationError(
+ "Can't read ", sizeof(T), " bytes from WKBBuffer with ", size_, " remaining");
+ }
+
+ return ReadUnchecked<T>();
+ }
+
+ template <typename T>
+ T ReadUnchecked() {
+ T out = ::arrow::util::SafeLoadAs<T>(data_);
+ data_ += sizeof(T);
+ size_ -= sizeof(T);
+ return out;
+ }
+
+ template <typename T>
+ T ByteSwap(T value) {
+ return ::arrow::bit_util::ByteSwap(value);
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ ::arrow::Status ReadGeometry(WKBBuffer* src) { return ReadGeometryInternal(src); }
+
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ const BoundingBox& Bounds() const { return box_; }
+
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ using GeometryTypeAndDimensions = std::pair<GeometryType, Dimensions>;
+
+ ::arrow::Status ReadGeometryInternal(WKBBuffer* src, bool record_wkb_type = true) {
+ ARROW_ASSIGN_OR_RAISE(uint8_t endian, src->ReadUInt8());
+#if defined(ARROW_LITTLE_ENDIAN)
+ bool swap = endian != 0x01;
+#else
+ bool swap = endian != 0x00;
+#endif
+
+ ARROW_ASSIGN_OR_RAISE(uint32_t wkb_geometry_type, src->ReadUInt32(swap));
+ ARROW_ASSIGN_OR_RAISE(auto geometry_type_and_dimensions,
+ ParseGeometryType(wkb_geometry_type));
+
+ // Keep track of geometry types encountered if at the top level
+ if (record_wkb_type) {
+ geospatial_types_.insert(static_cast<int32_t>(wkb_geometry_type));
+ }
+
+ switch (geometry_type_and_dimensions.first) {
+ case GeometryType::POINT:
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, 1, swap));
+ break;
+
+ case GeometryType::LINESTRING: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ break;
+ }
+ case GeometryType::POLYGON: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_coords, src->ReadUInt32(swap));
+ ARROW_RETURN_NOT_OK(
+ ReadSequence(src, geometry_type_and_dimensions.second, n_coords, swap));
+ }
+ break;
+ }
+
+ // These are all encoded the same in WKB, even though this encoding would
+ // allow for parts to be of a different geometry type or different dimensions.
+ // For the purposes of bounding, this does not cause us problems.
+ case GeometryType::MULTIPOINT:
+ case GeometryType::MULTILINESTRING:
+ case GeometryType::MULTIPOLYGON:
+ case GeometryType::GEOMETRYCOLLECTION: {
+ ARROW_ASSIGN_OR_RAISE(uint32_t n_parts, src->ReadUInt32(swap));
+ for (uint32_t i = 0; i < n_parts; i++) {
+ ARROW_RETURN_NOT_OK(ReadGeometryInternal(src, /*record_wkb_type*/ false));
+ }
+ break;
+ }
+ }
+
+ return ::arrow::Status::OK();
+ }
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions dimensions, uint32_t n_coords,
+ bool swap) {
+ switch (dimensions) {
+ case Dimensions::XY:
+ return src->ReadDoubles<BoundingBox::XY>(
+ n_coords, swap, [&](BoundingBox::XY coord) { box_.UpdateXY(coord); });
+ case Dimensions::XYZ:
+ return src->ReadDoubles<BoundingBox::XYZ>(
+ n_coords, swap, [&](BoundingBox::XYZ coord) { box_.UpdateXYZ(coord); });
+ case Dimensions::XYM:
+ return src->ReadDoubles<BoundingBox::XYM>(
+ n_coords, swap, [&](BoundingBox::XYM coord) { box_.UpdateXYM(coord); });
+ case Dimensions::XYZM:
+ return src->ReadDoubles<BoundingBox::XYZM>(
+ n_coords, swap, [&](BoundingBox::XYZM coord) { box_.UpdateXYZM(coord); });
+ default:
+ return ::arrow::Status::Invalid("Unknown dimensions");
+ }
+ }
+
+ static ::arrow::Result<GeometryTypeAndDimensions> ParseGeometryType(
+ uint32_t wkb_geometry_type) {
+ // The number 1000 can be used because WKB geometry types are constructed
+ // on purpose such that this relationship is true (e.g., LINESTRING ZM maps
+ // to 3002).
+ uint32_t geometry_type_component = wkb_geometry_type % 1000;
+ uint32_t dimensions_component = wkb_geometry_type / 1000;
+
+ auto min_geometry_type_value = static_cast<uint32_t>(GeometryType::POINT);
+ auto max_geometry_type_value =
+ static_cast<uint32_t>(GeometryType::GEOMETRYCOLLECTION);
+ auto min_dimension_value = static_cast<uint32_t>(Dimensions::XY);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970048200
##########
cpp/src/parquet/geometry_util_internal.cc:
##########
@@ -0,0 +1,234 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_util_internal.h"
+
+#include "arrow/result.h"
+#include "arrow/util/endian.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/ubsan.h"
+
+namespace parquet::geometry {
+
+/// \brief Object to keep track of the low-level consumption of a well-known binary
+/// geometry
+///
+/// Briefly, ISO well-known binary supported by the Parquet spec is an endian byte
+/// (0x01 or 0x00), followed by geometry type + dimensions encoded as a (uint32_t),
+/// followed by geometry-specific data. Coordinate sequences are represented by a
+/// uint32_t (the number of coordinates) plus a sequence of doubles (number of coordinates
+/// multiplied by the number of dimensions).
+class WKBBuffer {
+ public:
+ WKBBuffer() : data_(NULLPTR), size_(0) {}
+ WKBBuffer(const uint8_t* data, int64_t size) : data_(data), size_(size) {}
Review Comment:
I'm happy to do anything here (`ByteArray` might also be an option?). Perhaps if this is made external we can rethink the interface if needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969530501
##########
cpp/src/parquet/column_writer_test.cc:
##########
@@ -399,6 +399,26 @@ class TestPrimitiveWriter : public PrimitiveTypedTest<TestType> {
return metadata_accessor->key_value_metadata();
}
+ std::unique_ptr<ColumnChunkMetaData> metadata_accessor() {
+ // Metadata accessor must be created lazily.
+ // This is because the ColumnChunkMetaData semantics dictate the metadata object is
+ // complete (no changes to the metadata buffer can be made after instantiation)
+ return ColumnChunkMetaData::Make(metadata_->contents(), this->descr_);
+ }
+
+ EncodedStatistics metadata_encoded_stats() { return metadata_stats()->Encode(); }
+
+ std::shared_ptr<Statistics> metadata_stats() {
+ return metadata_accessor()->statistics();
+ }
+
+ std::shared_ptr<GeospatialStatistics> metadata_geometry_stats() {
+ ApplicationVersion app_version(this->writer_properties_->created_by());
+ auto metadata_accessor = ColumnChunkMetaData::Make(
+ metadata_->contents(), this->descr_, default_reader_properties(), &app_version);
Review Comment:
That was a result of the rather extensive rebase on top of the size statistics change 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1969532017
##########
cpp/src/parquet/arrow/schema_internal.cc:
##########
@@ -110,8 +111,62 @@ Result<std::shared_ptr<ArrowType>> MakeArrowTimestamp(const LogicalType& logical
}
}
+Result<std::string> MakeGeoArrowCrsMetadata(
+ const std::string& crs,
+ const std::shared_ptr<const ::arrow::KeyValueMetadata>& metadata) {
+ std::string srid_prefix{"srid:"};
+ std::string projjson_prefix{"projjson:"};
+
+ if (crs.empty()) {
+ return R"("crs": "OGC:CRS84", "crs_type": "authority_code")";
+ } else if (crs.rfind(srid_prefix, 0) == 0) {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1970767257
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
+
+ /// \brief Update these statistics with previously generated statistics
+ void Decode(const EncodedGeospatialStatistics& encoded);
+
+ /// \brief The minimum encountered value in the X dimension, or Inf if no X values were
+ /// encountered.
+ ///
+ /// The Parquet definition allows for "wrap around" bounds where xmin > xmax. In this
+ /// case, these bounds represent the union of the intervals [xmax, Inf] and [-Inf,
+ /// xmin]. This implementation does not yet generate these types of bounds but they may
+ /// be encountered in files written by other readers.
+ double GetXMin() const;
Review Comment:
It depends on whether `Inf/-Inf` is a common practice for bbox or a helper function is needed for users to understand that the bbox is empty.
BTW, I think C++ and Java implementations should be consistent: https://github.com/apache/parquet-java/pull/2971/files#diff-c47987281a33dfa16169ca9dc1e2011db9bf18bb8fbd27601207b5697d6f0e77
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974507677
##########
cpp/src/parquet/types.h:
##########
@@ -446,6 +467,30 @@ class PARQUET_EXPORT Float16LogicalType : public LogicalType {
Float16LogicalType() = default;
};
+class PARQUET_EXPORT GeometryLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(std::string crs = "");
+
+ const std::string& crs() const;
+
+ private:
+ GeometryLogicalType() = default;
+};
+
+class PARQUET_EXPORT GeographyLogicalType : public LogicalType {
+ public:
+ static std::shared_ptr<const LogicalType> Make(
+ std::string crs = "", LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm =
+ EdgeInterpolationAlgorithm::SPHERICAL);
+
+ const std::string& crs() const;
+ LogicalType::EdgeInterpolationAlgorithm::algorithm algorithm() const;
+ const char* algorithm_name() const;
Review Comment:
I didn't add one because I couldn't find a public `TimeUnitToString` and an `algorithm_name()` method is a little nicer (but I'm happy to do either!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974522330
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
Review Comment:
This now checks all the values to check the "setness" 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974521566
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1792,6 +1831,12 @@ Status TypedColumnWriterImpl<DType>::WriteArrowDictionary(
page_statistics_->IncrementNullCount(num_chunk_levels - non_null_count);
page_statistics_->IncrementNumValues(non_null_count);
page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false);
+
+ if constexpr (std::is_same<T, ByteArray>::value) {
+ if (chunk_geometry_statistics_ != nullptr) {
+ chunk_geometry_statistics_->Update(*referenced_dictionary);
Review Comment:
I updated this to throw (but if you try to write a dictionary here it fails at the schema conversion step anyway)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975817298
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
Review Comment:
These now accept a `span`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #45459:
URL: https://github.com/apache/arrow/pull/45459#issuecomment-2691485598
@emkornfield @pitrou @wgtmac Thank you for your reviews! I think I managed to get the spirit of all of your comments throughout the diff, although I am sure I missed something and obviously feel free to let me know what it was! I think the outstanding issues are:
- I merged the changes in from the branch I had been using to write the test files, so as is, this PR is fully functional for arbitrary cases of `write_parquet(geopandas_data_frame.to_arrow())` or `write_parquet(duckdb_rel.to_arrow_table())` or `write_parquet(sedona.spark.dataframe_arrow(spark_df))` with the extension type registered. I made the JSON dependency optional and separated the code so that it's easy to remove if that's not something we want. I'm totally game to remove that...we can do enough without parsing JSON that all of the reading/writing code is tested and we can solve the full utility later on (e.g., through a canonical extension type).
- I re-read the "crs" section of the final spec and remembered very late that the `projjson:some_file_metadata_field` is a *recommendation* and not a requirement. It's obviously much easier to just dump GeoArrow's `"crs": ...` to Parquet's `crs` and vice versa and it would be very easy to convince me not to implement the recommendation (which would allow the removal of the `GeoCrsContext` from the `ArrowWriterProperties` and the `KeyValueMetadata` reference from the node context thing in the other direction). Alternatively, I'm happy to remove all non-lon/lat CRS handling (which still allows all of the read/write code to be tested) and figure that out later.
- I am wondering if the R CI failures are because I added a `shared_ptr` to the `ArrowWriterProperties`. Happy to work through that if we decide we still want it there!
(Or feel free to add something to this list!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975794192
##########
cpp/src/parquet/geometry_statistics.h:
##########
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet {
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeospatialStatistics for the specific definition
+/// of field values.
+class PARQUET_EXPORT EncodedGeospatialStatistics {
+ public:
+ static constexpr double kInf = std::numeric_limits<double>::infinity();
+
+ EncodedGeospatialStatistics() = default;
+ EncodedGeospatialStatistics(const EncodedGeospatialStatistics&) = default;
+ EncodedGeospatialStatistics(EncodedGeospatialStatistics&&) = default;
+ EncodedGeospatialStatistics& operator=(const EncodedGeospatialStatistics&) = default;
+
+ double xmin{kInf};
+ double xmax{-kInf};
+ double ymin{kInf};
+ double ymax{-kInf};
+ double zmin{kInf};
+ double zmax{-kInf};
+ double mmin{kInf};
+ double mmax{-kInf};
+ std::vector<int32_t> geospatial_types;
+
+ bool has_z() const { return (zmax - zmin) != -kInf; }
+
+ bool has_m() const { return (mmax - mmin) != -kInf; }
+
+ bool is_set() const { return !geospatial_types.empty(); }
+};
+
+class GeospatialStatisticsImpl;
+
+/// \brief Base type for computing geospatial column statistics while writing a file
+class PARQUET_EXPORT GeospatialStatistics {
+ public:
+ GeospatialStatistics();
+ explicit GeospatialStatistics(std::unique_ptr<GeospatialStatisticsImpl> impl);
+ explicit GeospatialStatistics(const EncodedGeospatialStatistics& encoded);
+ GeospatialStatistics(GeospatialStatistics&&);
+
+ ~GeospatialStatistics();
+
+ /// \brief Return true if bounds, geometry types, and validity are identical
+ bool Equals(const GeospatialStatistics& other) const;
+
+ /// \brief Update these statistics based on previously calculated or decoded statistics
+ void Merge(const GeospatialStatistics& other);
+
+ /// \brief Update these statistics based on values
+ void Update(const ByteArray* values, int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ void UpdateSpaced(const ByteArray* values, const uint8_t* valid_bits,
+ int64_t valid_bits_offset, int64_t num_spaced_values,
+ int64_t num_values, int64_t null_count);
+
+ /// \brief Update these statistics based on the non-null elements of values
+ ///
+ /// Currently, BinaryArray and LargeBinaryArray input is supported.
+ void Update(const ::arrow::Array& values);
+
+ /// \brief Return these statistics to an empty state
+ void Reset();
+
+ /// \brief Encode the statistics for serializing to Thrift
+ ///
+ /// If invalid WKB was encountered, empty encoded statistics are returned
+ /// (such that is_set() returns false and they should not be written).
+ EncodedGeospatialStatistics Encode() const;
+
+ /// \brief Returns true if all WKB encountered was valid or false otherwise
+ bool is_valid() const;
+
+ std::shared_ptr<GeospatialStatistics> clone() const;
Review Comment:
I think we added it initially for parity with `Statistics`, but I removed it since I'm not sure where it's needed (it could be added later if it is, or somebody can just `Encode()` then `Decode()` to implement it themselves)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1974547592
##########
cpp/src/parquet/geometry_statistics.cc:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/geometry_statistics.h"
+#include <memory>
+
+#include "arrow/array.h"
+#include "arrow/type.h"
+#include "arrow/util/bit_run_reader.h"
+#include "arrow/util/logging.h"
+#include "parquet/exception.h"
+#include "parquet/geometry_util_internal.h"
+
+using arrow::util::SafeLoad;
+
+namespace parquet {
+
+class GeospatialStatisticsImpl {
+ public:
+ GeospatialStatisticsImpl() = default;
+ GeospatialStatisticsImpl(const GeospatialStatisticsImpl&) = default;
+
+ bool Equals(const GeospatialStatisticsImpl& other) const {
+ if (is_valid_ != other.is_valid_) {
+ return false;
+ }
+
+ if (!is_valid_ && !other.is_valid_) {
+ return true;
+ }
+
+ auto geospatial_types = bounder_.GeometryTypes();
+ auto other_geospatial_types = other.bounder_.GeometryTypes();
+ if (geospatial_types.size() != other_geospatial_types.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < geospatial_types.size(); i++) {
+ if (geospatial_types[i] != other_geospatial_types[i]) {
+ return false;
+ }
+ }
Review Comment:
I changed this and it's much nicer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975826854
##########
cpp/src/parquet/geometry_util_internal.h:
##########
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <limits>
+#include <sstream>
+#include <string>
+#include <unordered_set>
+
+#include "parquet/platform.h"
+
+namespace parquet::geometry {
+
+/// \brief Infinity, used to define bounds of empty bounding boxes
+constexpr double kInf = std::numeric_limits<double>::infinity();
+
+/// \brief Valid combinations of dimensions allowed by ISO well-known binary
+enum class Dimensions { XY = 0, XYZ = 1, XYM = 2, XYZM = 3, MIN = 0, MAX = 3 };
+
+/// \brief The supported set of geometry types allowed by ISO well-known binary
+enum class GeometryType {
+ POINT = 1,
+ LINESTRING = 2,
+ POLYGON = 3,
+ MULTIPOINT = 4,
+ MULTILINESTRING = 5,
+ MULTIPOLYGON = 6,
+ GEOMETRYCOLLECTION = 7,
+ MIN = 1,
+ MAX = 7
+};
+
+/// \brief A collection of intervals representing the encountered ranges of values
+/// in each dimension.
+struct BoundingBox {
+ using XY = std::array<double, 2>;
+ using XYZ = std::array<double, 3>;
+ using XYM = std::array<double, 3>;
+ using XYZM = std::array<double, 4>;
+
+ BoundingBox(const XYZM& mins, const XYZM& maxes) : min(mins), max(maxes) {}
+ BoundingBox() : min{kInf, kInf, kInf, kInf}, max{-kInf, -kInf, -kInf, -kInf} {}
+
+ BoundingBox(const BoundingBox& other) = default;
+ BoundingBox& operator=(const BoundingBox&) = default;
+
+ /// \brief Update the X and Y bounds to ensure these bounds contain coord
+ void UpdateXY(const XY& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and Z bounds to ensure these bounds contain coord
+ void UpdateXYZ(const XYZ& coord) { UpdateInternal(coord); }
+
+ /// \brief Update the X, Y, and M bounds to ensure these bounds contain coord
+ void UpdateXYM(const XYM& coord) {
+ min[0] = std::min(min[0], coord[0]);
+ min[1] = std::min(min[1], coord[1]);
+ min[3] = std::min(min[3], coord[2]);
+ max[0] = std::max(max[0], coord[0]);
+ max[1] = std::max(max[1], coord[1]);
+ max[3] = std::max(max[3], coord[2]);
+ }
+
+ /// \brief Update the X, Y, Z, and M bounds to ensure these bounds contain coord
+ void UpdateXYZM(const XYZM& coord) { UpdateInternal(coord); }
+
+ /// \brief Reset these bounds to an empty state such that they contain no coordinates
+ void Reset() {
+ for (int i = 0; i < 4; i++) {
+ min[i] = kInf;
+ max[i] = -kInf;
+ }
+ }
+
+ /// \brief Update these bounds such they also contain other
+ void Merge(const BoundingBox& other) {
+ for (int i = 0; i < 4; i++) {
+ min[i] = std::min(min[i], other.min[i]);
+ max[i] = std::max(max[i], other.max[i]);
+ }
+ }
+
+ std::string ToString() const {
+ std::stringstream ss;
+ ss << "BoundingBox [" << min[0] << " => " << max[0];
+ for (int i = 1; i < 4; i++) {
+ ss << ", " << min[i] << " => " << max[i];
+ }
+
+ ss << "]";
+
+ return ss.str();
+ }
+
+ XYZM min;
+ XYZM max;
+
+ private:
+ // This works for XY, XYZ, and XYZM
+ template <typename Coord>
+ void UpdateInternal(Coord coord) {
+ static_assert(coord.size() <= 4);
+
+ for (size_t i = 0; i < coord.size(); i++) {
+ min[i] = std::min(min[i], coord[i]);
+ max[i] = std::max(max[i], coord[i]);
+ }
+ }
+};
+
+inline bool operator==(const BoundingBox& lhs, const BoundingBox& rhs) {
+ return lhs.min == rhs.min && lhs.max == rhs.max;
+}
+
+class WKBBuffer;
+
+/// \brief Accumulate a BoundingBox and geometry types based on zero or more well-known
+/// binary blobs
+class PARQUET_EXPORT WKBGeometryBounder {
+ public:
+ WKBGeometryBounder() = default;
+ WKBGeometryBounder(const WKBGeometryBounder&) = default;
+
+ /// \brief Accumulate the bounds of a serialized well-known binary geometry
+ ///
+ /// Returns SerializationError for any parse errors encountered. Bounds for
+ /// any encountered coordinates are accumulated and the geometry type of
+ /// the geometry is added to the internal geometry type list.
+ ///
+ /// Note that this method is NOT appropriate for bounding a GEOGRAPHY,
+ /// whose bounds are not a function purely of the vertices. Geography bounding
+ /// is not yet implemented.
+ ::arrow::Status ReadGeometry(const uint8_t* data, int64_t size);
+
+ /// \brief Accumulate the bounds of a previously-calculated BoundingBox
+ void ReadBox(const BoundingBox& box) { box_.Merge(box); }
+
+ /// \brief Accumulate a previously-calculated list of geometry types
+ void ReadGeometryTypes(const std::vector<int32_t>& geospatial_types) {
+ geospatial_types_.insert(geospatial_types.begin(), geospatial_types.end());
+ }
+
+ /// \brief Retrieve the accumulated bounds
+ const BoundingBox& Bounds() const { return box_; }
+
+ /// \brief Retrieve the accumulated geometry types
+ std::vector<int32_t> GeometryTypes() const {
+ std::vector<int32_t> out(geospatial_types_.begin(), geospatial_types_.end());
+ std::sort(out.begin(), out.end());
+ return out;
+ }
+
+ /// \brief Reset the internal bounds and geometry types list to an empty state
+ void Reset() {
+ box_.Reset();
+ geospatial_types_.clear();
+ }
+
+ private:
+ BoundingBox box_;
+ std::unordered_set<int32_t> geospatial_types_;
+
+ ::arrow::Status ReadGeometryInternal(WKBBuffer* src, bool record_wkb_type);
+
+ ::arrow::Status ReadSequence(WKBBuffer* src, Dimensions dimensions, uint32_t n_coords,
+ bool swap);
Review Comment:
Yes, these are called from `ReadGeometry()`. I briefly experimented with putting them in an anonymous namespace in the .cc file but I didn't like the look of it (required passing some of the private fields of the WKBBounder to the functions), but I'd be happy to try that again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975828416
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
+
+ BoundingBox box_xyzm({-1, -2, -3, -4}, {1, 2, 3, 4});
+ BoundingBox box_xy({-10, -20, kInf, kInf}, {10, 20, -kInf, -kInf});
+ BoundingBox box_xyz({kInf, kInf, -30, kInf}, {-kInf, -kInf, 30, -kInf});
+ BoundingBox box_xym({kInf, kInf, kInf, -40}, {-kInf, -kInf, -kInf, 40});
+
+ box_xyzm.Merge(box_xy);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -3, -4}, {10, 20, 3, 4}));
+
+ box_xyzm.Merge(box_xyz);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -4}, {10, 20, 30, 4}));
+
+ box_xyzm.Merge(box_xym);
+ EXPECT_EQ(box_xyzm, BoundingBox({-10, -20, -30, -40}, {10, 20, 30, 40}));
+
+ box_xyzm.Reset();
+ EXPECT_EQ(box_xyzm, BoundingBox());
+}
+
+struct WKBTestCase {
+ WKBTestCase() = default;
+ WKBTestCase(GeometryType x, Dimensions y, const std::vector<uint8_t>& z,
+ const std::vector<double>& box_values = {})
+ : geometry_type(x), dimensions(y), wkb(z) {
+ std::array<double, 4> mins = {kInf, kInf, kInf, kInf};
+ std::array<double, 4> maxes{-kInf, -kInf, -kInf, -kInf};
+
+ if (dimensions == Dimensions::XYM) {
+ mins = {box_values[0], box_values[1], kInf, box_values[2]};
+ maxes = {box_values[3], box_values[4], -kInf, box_values[5]};
+ } else {
+ size_t coord_size = box_values.size() / 2;
+ for (uint32_t i = 0; i < coord_size; i++) {
+ mins[i] = box_values[i];
+ maxes[i] = box_values[coord_size + i];
+ }
+ }
+
+ box = BoundingBox(mins, maxes);
+ }
+ WKBTestCase(const WKBTestCase& other) = default;
Review Comment:
I think I removed all of these!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r1975827290
##########
cpp/src/parquet/geometry_util_internal_test.cc:
##########
@@ -0,0 +1,455 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock.h>
+#include <gtest/gtest.h>
+#include <cmath>
+#include <cstring>
+
+#include "arrow/testing/gtest_util.h"
+
+#include "parquet/geometry_util_internal.h"
+#include "parquet/test_util.h"
+
+namespace parquet::geometry {
+
+TEST(TestGeometryUtil, TestBoundingBox) {
+ BoundingBox box;
+ EXPECT_EQ(box, BoundingBox({kInf, kInf, kInf, kInf}, {-kInf, -kInf, -kInf, -kInf}));
+ EXPECT_EQ(box.ToString(),
+ "BoundingBox [inf => -inf, inf => -inf, inf => -inf, inf => -inf]");
Review Comment:
I added the dimensions to the output here and it's much nicer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059450855
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
FYI: We can remove this `if(NOT ARROW_JSON)` with the suggested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2057973315
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
Ha... so we've had this option for ~8 years and it was never tested or documented anywhere. I've opened https://github.com/apache/arrow/issues/46219 so that we may eventually remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #45459:
URL: https://github.com/apache/arrow/pull/45459#discussion_r2059533673
##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -260,6 +263,14 @@ endif()
if(NOT PARQUET_MINIMAL_DEPENDENCY)
list(APPEND PARQUET_SHARED_LINK_LIBS arrow_shared)
+ # Add RapidJSON libraries
+ if(NOT ARROW_JSON)
Review Comment:
Done!
##########
cpp/src/parquet/geospatial/statistics.h:
##########
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+#include "parquet/platform.h"
+#include "parquet/types.h"
+
+namespace parquet::geospatial {
+
+/// \brief The maximum number of dimensions represented by a geospatial type
+/// (i.e., X, Y, Z, and M)
+static constexpr int kMaxDimensions = 4;
+
+/// \brief NaN, used to represent bounds for which predicate pushdown cannnot
+/// be applied (e.g., because a writer did not provide bounds for a given dimension)
+constexpr double kNaN = std::numeric_limits<double>::quiet_NaN();
+
+/// \brief Structure represented encoded statistics to be written to and read from Parquet
+/// serialized metadata.
+///
+/// See the Parquet Thrift definition and GeoStatistics for the specific definition
+/// of field values.
+struct PARQUET_EXPORT EncodedGeoStatistics {
+ bool xy_bounds_present{};
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org