You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/07/03 12:12:01 UTC
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24540
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.
Architecture:
The C++ implementation is layered into five concerns:
- wkb-format.h: Low-level WKB binary parsing (header
validation, coordinate extraction, type mapping). Assumes
little-endian byte order throughout.
- wkb-serialization: Reads WKB bytes into Boost.Geometry
model types (point2d, linestring2d, polygon2d, etc.) and
writes them back. Also provides bounding-box computation
directly from raw WKB without full deserialization.
- geometry-wrapper-wkb: High-level wrapper that holds one
geometry instance (tagged union style via OGCType enum).
Provides FromWkb/FromWkt/FromCoordinates deserialization,
ToWkb/ToWkt serialization, and geometry property methods
(Area, Length, Distance, NumPoints, IsSimple, etc.) that
delegate to Boost.Geometry algorithms.
- relation-wrapper-wkb: Template-based spatial predicate
evaluator. Uses a policy pattern (ContainsPredicate,
IntersectsPredicate, etc.) with double dispatch over
LHS/RHS geometry types.
- geospatial-functions-wkb-ir.cc: Thin entry points
registered as Impala BUILTINs. These parse UDF arguments,
delegate to the wrapper/serialization layers above, and
return Impala UDF result types.
Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
output formatting.
- common.h: Boost.Geometry type aliases and OGC type
constants.
On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.
Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/formatted-double.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
21 files changed, 2,945 insertions(+), 367 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/1
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 15: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14086/
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 23:21:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24228/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 14:30:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23172/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 14:51:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,389 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/12
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 13:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/bin-utils.h@27
PS13, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS13, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS13, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@436
PS13, Line 436: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@438
PS13, Line 438: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:36:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 1:
(4 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@453
PS1, Line 453: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@462
PS1, Line 462: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/utils.h@28
PS1, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/1/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/1/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS1, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 12:14:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#14).
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,388 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/14
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 14:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/bin-utils.h@27
PS14, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS14, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS14, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@436
PS14, Line 436: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@438
PS14, Line 438: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:17:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Also changes orientation of polygons to use the more common CCW
winding.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/boost-common.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
M be/src/exprs/geo/wkb-serialization-test.cc
M be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
21 files changed, 2,389 insertions(+), 79 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/16
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 16:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/bin-utils.h@27
PS16, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS16, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS16, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@431
PS16, Line 431: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@433
PS16, Line 433: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 14:11:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Co-Authored-By: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
17 files changed, 2,369 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/6
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24540/8/be/src/exprs/geo/relation-wrapper-wkb-test.cc
File be/src/exprs/geo/relation-wrapper-wkb-test.cc:
http://gerrit.cloudera.org:8080/#/c/24540/8/be/src/exprs/geo/relation-wrapper-wkb-test.cc@33
PS8, Line 33: static polygon2d MakeSquare(double x0, double y0, double x1, double y1) {
: polygon2d poly;
: bg::append(poly.outer(), P(x0, y0));
: bg::append(poly.outer(), P(x1, y0));
: bg::append(poly.outer(), P(x1, y1));
: bg::append(poly.outer(), P(x0, y1));
: bg::append(poly.outer(), P(x0, y0));
: bg::correct(poly);
: return poly;
: }
Shouldn't it be wound clockwise (assuming x1 > x0 and y1 > y0)?
Also, a rectangle, not a square.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 10:26:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 2:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS2, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS2, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/utils.h@28
PS2, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS2, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS2, Line 433: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS2, Line 435: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 18:40:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23080/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 12:37:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 6:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/bin-utils.h@27
PS6, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS6, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS6, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS6, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS6, Line 435: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@437
PS6, Line 437: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 14:32:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Balazs Hevele, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#8).
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,371 insertions(+), 64 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/8
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 7:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/bin-utils.h@27
PS7, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS7, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS7, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS7, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS7, Line 435: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@437
PS7, Line 437: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:34:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#2).
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.
Architecture:
The C++ implementation is layered into five concerns:
- wkb-format.h: Low-level WKB binary parsing (header
validation, coordinate extraction, type mapping). Assumes
little-endian byte order throughout.
- wkb-serialization: Reads WKB bytes into Boost.Geometry
model types (point2d, linestring2d, polygon2d, etc.) and
writes them back. Also provides bounding-box computation
directly from raw WKB without full deserialization.
- geometry-wrapper-wkb: High-level wrapper that holds one
geometry instance (tagged union style via OGCType enum).
Provides FromWkb/FromWkt/FromCoordinates deserialization,
ToWkb/ToWkt serialization, and geometry property methods
(Area, Length, Distance, NumPoints, IsSimple, etc.) that
delegate to Boost.Geometry algorithms.
- relation-wrapper-wkb: Template-based spatial predicate
evaluator. Uses a policy pattern (ContainsPredicate,
IntersectsPredicate, etc.) with double dispatch over
LHS/RHS geometry types.
- geospatial-functions-wkb-ir.cc: Thin entry points
registered as Impala BUILTINs. These parse UDF arguments,
delegate to the wrapper/serialization layers above, and
return Impala UDF result types.
Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
output formatting.
- common.h: Boost.Geometry type aliases and OGC type
constants.
On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.
Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,699 insertions(+), 372 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/2
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 4:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/bin-utils.h@27
PS4, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS4, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS4, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS4, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS4, Line 433: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS4, Line 435: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:14:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#13).
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-extra.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
20 files changed, 2,396 insertions(+), 80 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/13
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24018/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Sep 2026 20:18:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 13:
ps13 rebases to a newer state of parent review https://gerrit.cloudera.org/#/c/24534/
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:31:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14086/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:52:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24099/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:37:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Balazs Hevele, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#7).
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,371 insertions(+), 64 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/7
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23085/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:00:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#3).
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.
Architecture:
The C++ implementation is layered into five concerns:
- wkb-format.h: Low-level WKB binary parsing (header
validation, coordinate extraction, type mapping). Assumes
little-endian byte order throughout.
- wkb-serialization: Reads WKB bytes into Boost.Geometry
model types (point2d, linestring2d, polygon2d, etc.) and
writes them back. Also provides bounding-box computation
directly from raw WKB without full deserialization.
- geometry-wrapper-wkb: High-level wrapper that holds one
geometry instance (tagged union style via OGCType enum).
Provides FromWkb/FromWkt/FromCoordinates deserialization,
ToWkb/ToWkt serialization, and geometry property methods
(Area, Length, Distance, NumPoints, IsSimple, etc.) that
delegate to Boost.Geometry algorithms.
- relation-wrapper-wkb: Template-based spatial predicate
evaluator. Uses a policy pattern (ContainsPredicate,
IntersectsPredicate, etc.) with double dispatch over
LHS/RHS geometry types.
- geospatial-functions-wkb-ir.cc: Thin entry points
registered as Impala BUILTINs. These parse UDF arguments,
delegate to the wrapper/serialization layers above, and
return Impala UDF result types.
Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
output formatting.
- common.h: Boost.Geometry type aliases and OGC type
constants.
On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.
Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,700 insertions(+), 372 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/3
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 3:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS3, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS3, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/utils.h@28
PS3, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS3, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.
http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS3, Line 433: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS3, Line 435: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 18:58:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#15).
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.
Also changes orientation of polygons to use the more common CCW
winding.
Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
boost::geometry types and back, and hosts the geometry property methods
(Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
compiled into the codegen module and not stripped.
New geo files use the bg:: alias (= boost::geometry) to match common.h.
Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
boost has no empty-point concept) moved to geospatial-esri-specific-
overloads.test as an ESRI-only case, pending unified empty-point handling.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/boost-common.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
M be/src/exprs/geo/wkb-serialization-test.cc
M be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
21 files changed, 2,390 insertions(+), 80 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/15
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14085/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:52:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 12:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/bin-utils.h@27
PS12, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS12, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS12, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@458
PS12, Line 458: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@460
PS12, Line 460: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Sep 2026 19:59:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23087/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:35:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24540
to look at the new patch set (#4).
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
POC: Port native C++ geospatial functions to WKB mode
Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.
Architecture:
The C++ implementation is layered into five concerns:
- wkb-format.h: Low-level WKB binary parsing (header
validation, coordinate extraction, type mapping). Assumes
little-endian byte order throughout.
- wkb-serialization: Reads WKB bytes into Boost.Geometry
model types (point2d, linestring2d, polygon2d, etc.) and
writes them back. Also provides bounding-box computation
directly from raw WKB without full deserialization.
- geometry-wrapper-wkb: High-level wrapper that holds one
geometry instance (tagged union style via OGCType enum).
Provides FromWkb/FromWkt/FromCoordinates deserialization,
ToWkb/ToWkt serialization, and geometry property methods
(Area, Length, Distance, NumPoints, IsSimple, etc.) that
delegate to Boost.Geometry algorithms.
- relation-wrapper-wkb: Template-based spatial predicate
evaluator. Uses a policy pattern (ContainsPredicate,
IntersectsPredicate, etc.) with double dispatch over
LHS/RHS geometry types.
- geospatial-functions-wkb-ir.cc: Thin entry points
registered as Impala BUILTINs. These parse UDF arguments,
delegate to the wrapper/serialization layers above, and
return Impala UDF result types.
Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
output formatting.
- common.h: Boost.Geometry type aliases and OGC type
constants.
On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.
Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.
Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,603 insertions(+), 372 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/4
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23086/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:19:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:55:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24100/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 19:15:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 15:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:
http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/bin-utils.h@27
PS15, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:
http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS15, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS15, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:
http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@436
PS15, Line 436: addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@438
PS15, Line 438: addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:53:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )
Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24024/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:50:49 +0000
Gerrit-HasComments: No