[jira] [Created] (SEDONA-180) Make better use of catalyst analyzer/optimizer - expression properties and traits

["Kristin Cowalcijk (Jira)" <ji...@apache.org>]

Kristin Cowalcijk created SEDONA-180:
----------------------------------------

             Summary: Make better use of catalyst analyzer/optimizer - expression properties and traits
                 Key: SEDONA-180
                 URL: https://issues.apache.org/jira/browse/SEDONA-180
             Project: Apache Sedona
          Issue Type: Improvement
            Reporter: Kristin Cowalcijk


There are several improvements we can make to catalyst expressions defined by Apache Sedona. This issue describes three trivial ones.

h3. Implement {{foldable}} for constant folding optimization

There're lots of range queries where the query window can be folded as a constant geometry, such as the following query:

{code:sql}
SELECT * FROM some_table WHERE ST_Within(geom, ST_GeomFromText('POLYGON (...)'))
{code}

The physical plan is:
{code}
== Physical Plan ==
Project [_1#9 AS latitude#14, _2#10 AS longitude#15, st_point(_2#10, _1#9) AS geom#18]
+- Filter st_within(st_point(_2#10, _1#9), st_geomfromtext(POLYGON ((0 0, 100 0, 100 100, 0 100, 0 0))))
   +- <table to be scanned>
{code}

The query window {{ST_GeomFromText('POLYGON (...)')}} can be folded as a constant, since the argument passed to {{ST_GeomFromText}} is a constant, and {{ST_GeomFromText}} is deterministic. We can implement the {{foldable}} method of such expressions to allow the constant folding optimizer to fold them:

{code:scala}
override def foldable: Boolean = inputExpressions.forall(_.foldable)
{code}

Once we have {{foldable}} method properly implemented, the physical plan becomes:

{code}
== Physical Plan ==
Project [_1#9 AS latitude#14, _2#10 AS longitude#15, st_point(_2#10, _1#9) AS geom#18]
+- Filter st_within(st_point(_2#10, _1#9), [1,3,0,0,32,0,0,0,0,1,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,89,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,89,64,0,0,0,0,0,0,89,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,89,64,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])
   +- <table to be scanned>
{code}

The second argument passed to st_within became a constant value, which is the internal representation of the {{GeometryUDT}} value.

Folding constant query window expressions is an important step to implement spatial predicate push down, and it is also beneficial to query performance since it eliminates unnecessary repeated evaluations.

h3. Declaration of Input Types

Catalyst expression can mixin the trait {{ExpectsInputTypes}} and implement {{inputTypes}} method to make catalyst analyzer type check the inputs of your expressions. For example, the following query

{code:sql}
SELECT ST_Contains('POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))', ST_GeomFromText('POINT (10.0 20.0)')
{code}

will raise an exception at runtime, since the first argument of {{ST_Contains}} should be a geometry:

{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (kontinuation executor driver): java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
	at org.apache.spark.sql.sedona_sql.expressions.ST_Contains.eval(Predicates.scala:49)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	...
{code}

We can implement the {{inputTypes}} method for {{ST_Contains}} to declare that the input of this expression should be two geometries:

{code:scala}
override def inputTypes: Seq[AbstractDataType] = Seq(GeometryUDT, GeometryUDT)
{code}

Now spark will emit a more informative error message when analyzing this query:

{code}
org.apache.spark.sql.AnalysisException: cannot resolve 'st_contains('POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))', st_geomfromtext('POINT (10.0 20.0)'))' due to data type mismatch: argument 1 requires geometry type, however, ''POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))'' is of string type.; line 1 pos 7;
'Project [unresolvedalias( **org.apache.spark.sql.sedona_sql.expressions.ST_Contains$**  , None)]
+- OneRowRelation
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$7(CheckAnalysis.scala:212)
...
{code}


h3. Implicit Input Type Casting

Sometimes we want the arguments passed to our expressions being implicitly casted to declared types. For example, {{ST_PolygonFromEnvelope}} takes 4 arguments, which are the coordinates of the envelope. Currently it only accepts double or decimal values as arguments. If integers were passed to {{ST_PolygonFromEnvelope}}, it will raise a {{scala.MatchError}} exception because we have not handled such cases:

{code:sql}
SELECT ST_PolygonFromEnvelope(1 + 1, 2 + 2, 3 + 3, 4 + 4)
{code}

{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (kontinuation executor driver): scala.MatchError: 2 (of class java.lang.Integer)
	at org.apache.spark.sql.sedona_sql.expressions.ST_PolygonFromEnvelope.eval(Constructors.scala:363)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
{code}

We can make {{ST_PolygonFromEnvelope}} mixin trait {{ImplicitCastInputTypes}}, which tells the analyzer to implicitly cast input values when possible. We can confirm that the input values were casted by inspecting the query plan:

{code}
== Physical Plan ==
Project [st_polygonfromenvelope(2.0, 4.0, 6.0, 8.0) AS st_polygonfromenvelope((1 + 1), (2 + 2), (3 + 3), (4 + 4))#13]
+- *(1) Scan OneRowRelation[]
{code}

Additionally, with {{foldable}} method properly implemented, the entire expression could be folded as a constant:

{code}
== Physical Plan ==
*(1) Project [[1,3,0,0,32,0,0,0,0,1,0,0,0,5,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,16,64,0,0,0,0,0,0,0,64,0,0,0,0,0,0,32,64,0,0,0,0,0,0,24,64,0,0,0,0,0,0,32,64,0,0,0,0,0,0,24,64,0,0,0,0,0,0,16,64,0,0,0,0,0,0,0,64,0,0,0,0,0,0,16,64] AS st_polygonfromenvelope((1 + 1), (2 + 2), (3 + 3), (4 + 4))#13]
+- *(1) Scan OneRowRelation[]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)