You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "iGN5117 (via GitHub)" <gi...@apache.org> on 2023/06/28 21:32:06 UTC
[GitHub] [sedona] iGN5117 opened a new pull request, #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
iGN5117 opened a new pull request, #878:
URL: https://github.com/apache/sedona/pull/878
## Did you read the Contributor Guide?
- Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
## Is this PR related to a JIRA ticket?
- Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-314. The PR name follows the format `[SEDONA-XXX] my subject`.
## What changes were proposed in this PR?
- Support optimized join on ST_Hausdorff by adding case matching on all overloaded hausdorff distances
## How was this patch tested?
- Added new unit tests
## Did this PR include necessary documentation updates?
- Yes, I have updated the documentation update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] iGN5117 commented on a diff in pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "iGN5117 (via GitHub)" <gi...@apache.org>.
iGN5117 commented on code in PR #878:
URL: https://github.com/apache/sedona/pull/878#discussion_r1246970019
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] iGN5117 commented on a diff in pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "iGN5117 (via GitHub)" <gi...@apache.org>.
iGN5117 commented on code in PR #878:
URL: https://github.com/apache/sedona/pull/878#discussion_r1246972419
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
Have added candidates 1, 2, 5, 10. These yield result 100, 298, 688, 1258 respectively for a sample size of 100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #878:
URL: https://github.com/apache/sedona/pull/878
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] iGN5117 commented on a diff in pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "iGN5117 (via GitHub)" <gi...@apache.org>.
iGN5117 commented on code in PR #878:
URL: https://github.com/apache/sedona/pull/878#discussion_r1246951826
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
Sure, both intersects (<=) and non-intersects (<) return a count of 100 for this dataset. The polygon-point pairs are also exactly same since there is no pair with frechetDistance == 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #878:
URL: https://github.com/apache/sedona/pull/878#discussion_r1246912587
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
I wonder what the number of expected intersects is in these test cases? Can you report the numbers here?
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
@iGN5117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #878: [SEDONA-314] Support optimized join on ST_HausdorffDistance
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #878:
URL: https://github.com/apache/sedona/pull/878#discussion_r1246959588
##########
sql/common/src/test/scala/org/apache/sedona/sql/predicateJoinTestScala.scala:
##########
@@ -403,7 +403,41 @@ class predicateJoinTestScala extends TestBaseScala {
assert(distanceJoinDf.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
assert(distanceJoinDf.count() == expected)
})
+ }
+
+ it("Passed ST_HausdorffDistance in a spatial join") {
+ val sampleCount = 100
+ val distanceCandidates = Seq(1, 2)
+ val densityFrac = 0.6
+ val inputPoint = buildPointDf.limit(sampleCount).repartition(5)
+ val inputPolygon = buildPolygonDf.limit(sampleCount).repartition(3)
+ distanceCandidates.foreach(distance => {
+
+ //DensityFrac specified, <= distance
+ val expectedDensityIntersects = bruteForceDistanceJoinHausdorff(sampleCount, distance, densityFrac, true)
+ val distanceDensityIntersectsDF = inputPoint.alias("pointDF").join(inputPolygon.alias("polygonDF"), expr(s"ST_HausdorffDistance(pointDF.pointshape, polygonDF.polygonshape, $densityFrac) <= $distance"))
+ assert(distanceDensityIntersectsDF.queryExecution.sparkPlan.collect { case p: DistanceJoinExec => p }.size === 1)
+ assert(distanceDensityIntersectsDF.count() == expectedDensityIntersects)
Review Comment:
Can you add more distanceCandidates? Most importantly, add some candidates that lead to results > 100.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org