You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/07/03 12:12:01 UTC

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24540


Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.

Architecture:

The C++ implementation is layered into five concerns:

- wkb-format.h: Low-level WKB binary parsing (header
  validation, coordinate extraction, type mapping). Assumes
  little-endian byte order throughout.

- wkb-serialization: Reads WKB bytes into Boost.Geometry
  model types (point2d, linestring2d, polygon2d, etc.) and
  writes them back. Also provides bounding-box computation
  directly from raw WKB without full deserialization.

- geometry-wrapper-wkb: High-level wrapper that holds one
  geometry instance (tagged union style via OGCType enum).
  Provides FromWkb/FromWkt/FromCoordinates deserialization,
  ToWkb/ToWkt serialization, and geometry property methods
  (Area, Length, Distance, NumPoints, IsSimple, etc.) that
  delegate to Boost.Geometry algorithms.

- relation-wrapper-wkb: Template-based spatial predicate
  evaluator. Uses a policy pattern (ContainsPredicate,
  IntersectsPredicate, etc.) with double dispatch over
  LHS/RHS geometry types.

- geospatial-functions-wkb-ir.cc: Thin entry points
  registered as Impala BUILTINs. These parse UDF arguments,
  delegate to the wrapper/serialization layers above, and
  return Impala UDF result types.

Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
  polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
  ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
  output formatting.
- common.h: Boost.Geometry type aliases and OGC type
  constants.

On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.

Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/formatted-double.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
21 files changed, 2,945 insertions(+), 367 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 15: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14086/


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 23:21:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24228/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 14:30:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23172/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 14:51:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,389 insertions(+), 83 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/12
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 13:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/bin-utils.h@27
PS13, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS13, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/13/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS13, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@436
PS13, Line 436:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/13/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@438
PS13, Line 438:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:36:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 1:

(4 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@453
PS1, Line 453: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@462
PS1, Line 462: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/1/be/src/exprs/geo/utils.h@28
PS1, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/1/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/1/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS1, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 12:14:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#14).

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,388 insertions(+), 75 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/14
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 14:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/bin-utils.h@27
PS14, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS14, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/14/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS14, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@436
PS14, Line 436:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/14/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@438
PS14, Line 438:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:17:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Also changes orientation of polygons to use the more common CCW
winding.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/boost-common.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
M be/src/exprs/geo/wkb-serialization-test.cc
M be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
21 files changed, 2,389 insertions(+), 79 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/16
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 16:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/bin-utils.h@27
PS16, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS16, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/16/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS16, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@431
PS16, Line 431:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/16/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@433
PS16, Line 433:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 14:11:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Co-Authored-By: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
17 files changed, 2,369 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24540/8/be/src/exprs/geo/relation-wrapper-wkb-test.cc
File be/src/exprs/geo/relation-wrapper-wkb-test.cc:

http://gerrit.cloudera.org:8080/#/c/24540/8/be/src/exprs/geo/relation-wrapper-wkb-test.cc@33
PS8, Line 33: static polygon2d MakeSquare(double x0, double y0, double x1, double y1) {
            :   polygon2d poly;
            :   bg::append(poly.outer(), P(x0, y0));
            :   bg::append(poly.outer(), P(x1, y0));
            :   bg::append(poly.outer(), P(x1, y1));
            :   bg::append(poly.outer(), P(x0, y1));
            :   bg::append(poly.outer(), P(x0, y0));
            :   bg::correct(poly);
            :   return poly;
            : }
Shouldn't it be wound clockwise (assuming x1 > x0 and y1 > y0)?
Also, a rectangle, not a square.



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 10:26:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 2:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS2, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS2, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/2/be/src/exprs/geo/utils.h@28
PS2, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS2, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.


http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS2, Line 433:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/2/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS2, Line 435:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 18:40:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23080/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 12:37:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 6:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/bin-utils.h@27
PS6, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS6, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/6/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS6, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS6, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.


http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS6, Line 435:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/6/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@437
PS6, Line 437:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 14:32:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Balazs Hevele, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#8).

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,371 insertions(+), 64 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/8
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 7:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/bin-utils.h@27
PS7, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS7, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/7/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS7, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS7, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.


http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS7, Line 435:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/7/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@437
PS7, Line 437:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:34:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#2).

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.

Architecture:

The C++ implementation is layered into five concerns:

- wkb-format.h: Low-level WKB binary parsing (header
  validation, coordinate extraction, type mapping). Assumes
  little-endian byte order throughout.

- wkb-serialization: Reads WKB bytes into Boost.Geometry
  model types (point2d, linestring2d, polygon2d, etc.) and
  writes them back. Also provides bounding-box computation
  directly from raw WKB without full deserialization.

- geometry-wrapper-wkb: High-level wrapper that holds one
  geometry instance (tagged union style via OGCType enum).
  Provides FromWkb/FromWkt/FromCoordinates deserialization,
  ToWkb/ToWkt serialization, and geometry property methods
  (Area, Length, Distance, NumPoints, IsSimple, etc.) that
  delegate to Boost.Geometry algorithms.

- relation-wrapper-wkb: Template-based spatial predicate
  evaluator. Uses a policy pattern (ContainsPredicate,
  IntersectsPredicate, etc.) with double dispatch over
  LHS/RHS geometry types.

- geospatial-functions-wkb-ir.cc: Thin entry points
  registered as Impala BUILTINs. These parse UDF arguments,
  delegate to the wrapper/serialization layers above, and
  return Impala UDF result types.

Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
  polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
  ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
  output formatting.
- common.h: Boost.Geometry type aliases and OGC type
  constants.

On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.

Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,699 insertions(+), 372 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 4:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/bin-utils.h@27
PS4, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS4, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/4/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS4, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS4, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.


http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS4, Line 433:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/4/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS4, Line 435:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:14:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#13).

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-extra.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
20 files changed, 2,396 insertions(+), 80 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/13
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24018/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Sep 2026 20:18:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 13:

ps13 rebases to a newer state of parent review https://gerrit.cloudera.org/#/c/24534/


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:31:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14086/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:52:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24099/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:37:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Balazs Hevele, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#7).

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
M tests/custom_cluster/test_geospatial_library.py
18 files changed, 2,371 insertions(+), 64 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23085/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:00:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#3).

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.

Architecture:

The C++ implementation is layered into five concerns:

- wkb-format.h: Low-level WKB binary parsing (header
  validation, coordinate extraction, type mapping). Assumes
  little-endian byte order throughout.

- wkb-serialization: Reads WKB bytes into Boost.Geometry
  model types (point2d, linestring2d, polygon2d, etc.) and
  writes them back. Also provides bounding-box computation
  directly from raw WKB without full deserialization.

- geometry-wrapper-wkb: High-level wrapper that holds one
  geometry instance (tagged union style via OGCType enum).
  Provides FromWkb/FromWkt/FromCoordinates deserialization,
  ToWkb/ToWkt serialization, and geometry property methods
  (Area, Length, Distance, NumPoints, IsSimple, etc.) that
  delegate to Boost.Geometry algorithms.

- relation-wrapper-wkb: Template-based spatial predicate
  evaluator. Uses a policy pattern (ContainsPredicate,
  IntersectsPredicate, etc.) with double dispatch over
  LHS/RHS geometry types.

- geospatial-functions-wkb-ir.cc: Thin entry points
  registered as Impala BUILTINs. These parse UDF arguments,
  delegate to the wrapper/serialization layers above, and
  return Impala UDF result types.

Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
  polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
  ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
  output formatting.
- common.h: Boost.Geometry type aliases and OGC type
  constants.

On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.

Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/utils.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,700 insertions(+), 372 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 3:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS3, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS3, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/utils.h
File be/src/exprs/geo/utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/3/be/src/exprs/geo/utils.h@28
PS3, Line 28: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@31
PS3, Line 31: import org.apache.impala.hive.geospatial.esri.*;
star import is not allowed. If you do need it, add it to JAVA_STAR_IMPORT_ALLOW_LIST in bin/jenkins/critique-gerrit-review.py.


http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@433
PS3, Line 433:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/3/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@435
PS3, Line 435:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 18:58:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Balazs Hevele, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#15).

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................

POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Implement native C++ ST_ geospatial functions for WKB_EXPERIMENTAL mode
so they no longer fall back to the Java (ESRI) implementations. Builds on
the WKB serialization foundation (common.h / wkb-format.h /
wkb-serialization.*) added in the previous commit. Kept as a single
commit; it can be split further later if desired.

Also changes orientation of polygons to use the more common CCW
winding.

Adds:
- geometry-wrapper-wkb.{h,cc}: GeometryWrapperWkb -- deserializes WKB into
  boost::geometry types and back, and hosts the geometry property methods
  (Area/Length/Distance/Centroid/...).
- relation-wrapper-wkb.{h,cc} (+ relation-wrapper-wkb-test.cc): native
  relation predicates (Contains/Crosses/Disjoint/Equals/Intersects/
  Overlaps/Touches/Within), with C++ unit coverage.
- geospatial-functions-wkb-ir.cc: the _WKB IR entry points -- accessors,
  constructors, predicates, transformations, binning and properties.
- wkt.h: WKT read helpers; bin-utils.h: ST_Bin binning helpers.
- HiveEsriGeospatialBuiltins.addWkbNatives(): registers the natives in
  WKB_EXPERIMENTAL mode; functions not registered fall back to Java.
- impala-ir.cc include + scalar-expr-evaluator force-refs so the IR is
  compiled into the codegen module and not stripped.

New geo files use the bg:: alias (= boost::geometry) to match common.h.

Tests:
- relation-wrapper-wkb-test.cc: C++ coverage of the relation predicates.
- test_wkb_experimental_relations_table + geospatial-relations-table.test:
  native relations over functional.geom_relations_wkt.
- The existing WKB geospatial tests now exercise the native path.
- ST_NumPoints of an empty point (0 in ESRI; 1 in the WKB native, since
  boost has no empty-point concept) moved to geospatial-esri-specific-
  overloads.test as an ESRI-only case, pending unified empty-point handling.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Asssisted-by: Claude Opus 4.8
---
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/boost-common.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
M be/src/exprs/geo/wkb-format.h
M be/src/exprs/geo/wkb-serialization-test.cc
M be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-specific-overloads.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
21 files changed, 2,390 insertions(+), 80 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/15
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14085/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:52:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 12:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/bin-utils.h@27
PS12, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS12, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/12/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS12, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@458
PS12, Line 458:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/12/fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java@460
PS12, Line 460:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Sep 2026 19:59:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 4:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23087/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:35:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24540

to look at the new patch set (#4).

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................

POC: Port native C++ geospatial functions to WKB mode

Port of https://gerrit.cloudera.org/#/c/20602/ to
WKB_EXPERIMENTAL mode. Functions that previously ran as Java
UDFs in WKB mode now have native C++ implementations using
Boost.Geometry. Only 2D geometries are supported — 3D/4D
types (Z/M dimensions) are rejected at the WKB header
parsing level.

Architecture:

The C++ implementation is layered into five concerns:

- wkb-format.h: Low-level WKB binary parsing (header
  validation, coordinate extraction, type mapping). Assumes
  little-endian byte order throughout.

- wkb-serialization: Reads WKB bytes into Boost.Geometry
  model types (point2d, linestring2d, polygon2d, etc.) and
  writes them back. Also provides bounding-box computation
  directly from raw WKB without full deserialization.

- geometry-wrapper-wkb: High-level wrapper that holds one
  geometry instance (tagged union style via OGCType enum).
  Provides FromWkb/FromWkt/FromCoordinates deserialization,
  ToWkb/ToWkt serialization, and geometry property methods
  (Area, Length, Distance, NumPoints, IsSimple, etc.) that
  delegate to Boost.Geometry algorithms.

- relation-wrapper-wkb: Template-based spatial predicate
  evaluator. Uses a policy pattern (ContainsPredicate,
  IntersectsPredicate, etc.) with double dispatch over
  LHS/RHS geometry types.

- geospatial-functions-wkb-ir.cc: Thin entry points
  registered as Impala BUILTINs. These parse UDF arguments,
  delegate to the wrapper/serialization layers above, and
  return Impala UDF result types.

Supporting headers:
- wkt.h: WKT parsing via Boost.Geometry read_wkt(), with
  polygon correction.
- utils.h: Binning logic (ported from ESRI BinUtils.java),
  ring orientation checks, and polygon assembly from rings.
- formatted-double.h: Custom double type for controlled WKT
  output formatting.
- common.h: Boost.Geometry type aliases and OGC type
  constants.

On the frontend, HiveEsriGeospatialBuiltins is reorganized
so that in WKB mode, functions with C++ natives are no longer
registered as Java UDFs. A new addWkbNatives() method
registers them as BUILTINs with GEOMETRY type signatures.

Tests for 3D/4D functions (ST_Z, ST_M, ST_Is3D, etc.) are
moved to a separate HIVE_ESRI-only test file since they are
not applicable to the WKB native path.

Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
---
M be/CMakeLists.txt
M be/src/codegen/impala-ir.cc
M be/src/exprs/geo/CMakeLists.txt
A be/src/exprs/geo/bin-utils.h
M be/src/exprs/geo/common.h
A be/src/exprs/geo/geometry-wrapper-wkb.cc
A be/src/exprs/geo/geometry-wrapper-wkb.h
A be/src/exprs/geo/geospatial-functions-wkb-ir.cc
M be/src/exprs/geo/geospatial-functions.h
A be/src/exprs/geo/relation-wrapper-wkb-test.cc
A be/src/exprs/geo/relation-wrapper-wkb.cc
A be/src/exprs/geo/relation-wrapper-wkb.h
A be/src/exprs/geo/wkb-format.h
A be/src/exprs/geo/wkb-serialization-test.cc
A be/src/exprs/geo/wkb-serialization.cc
A be/src/exprs/geo/wkb-serialization.h
A be/src/exprs/geo/wkt.h
M be/src/exprs/scalar-expr-evaluator.cc
M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-high-dimension.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-relations-table.test
A testdata/workloads/functional-query/queries/QueryTest/geospatial-wkb-serialization.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
27 files changed, 4,603 insertions(+), 372 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/24540/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 3:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23086/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2026 19:19:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 10 Jul 2026 12:55:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24100/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 19:15:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 15:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/bin-utils.h
File be/src/exprs/geo/bin-utils.h:

http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/bin-utils.h@27
PS15, Line 27: // https://github.com/Esri/spatial-framework-for-hadoop/blob/7226df669cbfaaf1edbfac0461acd1af45e12b81/hive/src/main/java/com/esri/hadoop/hive/BinUtils.java
line too long (155 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc
File be/src/exprs/geo/geospatial-functions-wkb-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@511
PS15, Line 511: // TODO: could parse the point count directly from WKB bytes, avoiding full deserialization.
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/15/be/src/exprs/geo/geospatial-functions-wkb-ir.cc@519
PS15, Line 519: // TODO: could read the sub-geometry count directly from WKB bytes (offset 5-8 for multi types).
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java:

http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@436
PS15, Line 436:     addNativeWithGeomState(db, "st_LineString", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24540/15/fe/src/compat-hive-3/java/org/apache/impala/compat/GeospatialBuiltins.java@438
PS15, Line 438:     addNativeWithGeomState(db, "st_MultiPoint", "_WKB", false, Type.GEOMETRY, Type.STRING);
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:53:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-15164: Port native C++ geospatial functions to WKB mode

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24540 )

Change subject: POC IMPALA-15164: Port native C++ geospatial functions to WKB mode
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24024/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24540
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0047acebd4aaa774bf6bb8ec0618515d38ad6aae
Gerrit-Change-Number: 24540
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Sep 2026 06:50:49 +0000
Gerrit-HasComments: No