[jira] [Created] (SEDONA-66) Coalesce after spatial join results in dropped rows.

["James Kyle (Jira)" <ji...@apache.org>]

James Kyle created SEDONA-66:
--------------------------------

             Summary: Coalesce after spatial join results in dropped rows.
                 Key: SEDONA-66
                 URL: https://issues.apache.org/jira/browse/SEDONA-66
             Project: Apache Sedona
          Issue Type: Bug
    Affects Versions: 1.0.0
            Reporter: James Kyle
         Attachments: image-2021-09-30-11-41-43-840.png, image-2021-09-30-11-43-18-164.png

The code snippet we were using internal that reliably reproduced this error looks something like

 

`leftSpatialRDD.spatialJoinFlat(rightSpatialRDD, ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)`

 

repartition does _not_ drop rows and serves as a workaround..

 

Below is the summary of our internal conversation that potentially root caused the problem. I've bolded the final sentence which is the summary of the potential root cause. 

 

 
h2. ISSUE 1

 
{code:java}
   val spatialRDD = makeSpatial(ds)
    setSpatialPartitioner(spatialRDD)
 
    val joined = spatialRDD.spatialJoinFlat(spatialRDD, ALLOW_OVERLAP).coalesce(NUM_POST_JOIN_PARTITIONS)
    val joinedNew = spatialRDD.spatialJoinFlat(spatialRDD, ALLOW_OVERLAP)
 
{code}
 

zipPartitions, and within each partition, it does something like indexed/for loop element checking

!image-2021-09-30-11-41-43-840.png|width=819,height=454!

then looking at the DAG of executed code, there are something interesting , first is SparkSQL inject a Scan operator between Sedona's RDD operation and the following DataFrame operations (.coalesce), and zipPartitions and coalesce is put in the same stage, the colocated zipPartitions and coalesce means that, even the joined RDD from Sedona is supposed to have N partitions, it will still converge to X partitions where X is specified by coalesce, !image-2021-09-30-11-43-18-164.png|width=575,height=972!

assumption of self join, and the implementation of coalesce, selfjoin assumes that the partitions will be 1:1 mapped, however, coalesce is a wrapper of upstream iterators which doesn't guarantee the 1:1 mapped, *it is highly possible that the highlight* *match in the screenshot returns a lot of false* 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)