[Impala-ASF-CR] IMPALA-11745: Add Hive's ESRI geospatial functions as builtins

["Peter Rozsa (Code Review)" <ge...@cloudera.org>]

Peter Rozsa has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19425 )

Change subject: IMPALA-11745: Add Hive's ESRI geospatial functions as builtins
......................................................................

IMPALA-11745: Add Hive's ESRI geospatial functions as builtins

This change adds geospatial functions from Hive's ESRI library as builtin UDFs.
Plain Hive UDFs are imported without changes, but the generic and varargs
functions are handled differently; generic functions are added with all of the
combinations of their parameters (cartesian product of the parameters), and
varargs functions are unfolded as an nth parameter simple function. The varargs
function wrappers are generated at build time and they can be configured in
gen_geospatial_udf_wrappers.py. These additional steps are required because of
the limitations in Impala's UDFExecuter which could be further improved; in
this case, the additional wrapping/mapping steps could be removed.

A new backend flag was added to turn this feature on/off as "geospatial_library".
The default value is "NONE" which means no geospatial function gets registered
as builtin, "HIVE_ESRI" value enables this implementation.

Known limitations:
 - ST_MultiLineString, ST_MultiPolygon only works with the WKT overload
 - ST_Polygon supports a maximum of 6 pairs of coordinates
 - ST_MultiPoint, ST_LineStrin supports a maximum of 7 pairs of coordinates
 - ST_ConvexHull, ST_Union supports a maximum of 6 geoms

These limits can be increased in gen_geospatial_udf_wrappers.py and
HiveEsriGeospatialBuiltins.java

Tests:
 - test_geospatial_udfs.py added based on
   https://github.com/Esri/spatial-framework-for-hadoop/tree/master/hive/test

Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225
---
M be/src/common/global-flags.cc
M be/src/exprs/hive-udf-call.cc
M be/src/util/backend-gflag-util.cc
M common/function-registry/CMakeLists.txt
A common/function-registry/gen_geospatial_udf_wrappers.py
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
A fe/src/main/java/org/apache/impala/hive/executor/BinaryToBinaryHiveLegacyFunctionExtractor.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactory.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactoryImpl.java
A fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyFunctionExtractor.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/ImpalaDoubleWritable.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/hive/executor/TestHiveJavaFunctionFactory.java
M java/CMakeLists.txt
M java/shaded-deps/hive-exec/pom.xml
M testdata/datasets/README
A testdata/workloads/functional-query/queries/QueryTest/udf-esri-geospatial.test
A tests/custom_cluster/test_geospatial_udfs.py
27 files changed, 3,505 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/19425/10
-- 
To view, visit http://gerrit.cloudera.org:8080/19425
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225
Gerrit-Change-Number: 19425
Gerrit-PatchSet: 10
Gerrit-Owner: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>