You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/23 17:45:05 UTC
[GitHub] [spark] cloud-fan commented on a change in pull request #34053: [SPARK-36813][SQL][PYTHON] Propose an infrastructure of as-of join and imlement ps.merge_asof
cloud-fan commented on a change in pull request #34053:
URL: https://github.com/apache/spark/pull/34053#discussion_r715020883
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2122,6 +2125,68 @@ object RewriteIntersectAll extends Rule[LogicalPlan] {
}
}
+/**
+ * Replaces logical [[AsOfJoin]] operator using a combination of Join and Aggregate operator.
+ *
+ * Input Pseudo-Query:
+ * {{{
+ * SELECT * FROM left ASOF JOIN right ON (condition, as_of on(left.t, right.t), tolerance)
+ * }}}
+ *
+ * Rewritten Query:
+ * {{{
+ * SELECT left.*, __right__.*
+ * FROM (
+ * SELECT
+ * left.*,
+ * (
+ * SELECT MIN_BY(STRUCT(right.*), left.t - right.t)
+ * FROM right
+ * WHERE condition AND left.t >= right.t AND right.t >= left.t - tolerance
+ * ) as __right__
+ * FROM left
+ * )
+ * }}}
+ */
+object RewriteAsOfJoin extends Rule[LogicalPlan] {
Review comment:
nit: move it to a new file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org