You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2023/06/01 16:40:02 UTC
[GitHub] [spark] peter-toth commented on a diff in pull request #40744: [SPARK-24497][SQL] Support recursive SQL
peter-toth commented on code in PR #40744:
URL: https://github.com/apache/spark/pull/40744#discussion_r1213420095
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:
##########
@@ -714,6 +717,121 @@ case class UnionExec(children: Seq[SparkPlan]) extends SparkPlan {
copy(children = newChildren)
}
+/**
+ * The physical node for recursion.
+ *
+ * @param loopId The id of the loop.
+ * @param anchor The logical plan of the initial element of the loop.
+ * @param recursion The logical plan that describes the recursion with an [[UnionLoopRef]] node.
+ * @param output The output attributes of this loop.
+ * @param limit An optional limit that can be pushed down to the node to stop the loop earlier.
+ */
+case class UnionLoopExec(
+ loopId: Long,
+ @transient anchor: LogicalPlan,
+ @transient recursion: LogicalPlan,
+ override val output: Seq[Attribute],
+ limit: Option[Int] = None) extends LeafExecNode {
+
+ val levelLimit = conf.getConf(SQLConf.CTE_RECURSION_LEVEL_LIMIT)
+
+ // We store the initial and generated children of the loop in this buffer.
+ // Please note that all elements are cached because of performance reasons as they are needed for
+ // next iteration.
+ @transient private val unionDFs = mutable.ArrayBuffer.empty[DataFrame]
+
+ override def innerChildren: Seq[QueryPlan[_]] = Seq(anchor, recursion)
+
+ override lazy val metrics = Map(
+ "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+ private def cacheAndCount(plan: LogicalPlan, limit: Option[Long]) = {
+ val limitedPlan = limit.map(l => Limit(Literal(l.toInt), plan)).getOrElse(plan)
+ val df = Dataset.ofRows(session, limitedPlan).persist()
Review Comment:
`repartition()` seems to be good option to truncate RDD lineage and decrease task sizes to avoid stack overflow. I added it as the default cache mode in https://github.com/apache/spark/pull/40744/commits/02f527de85fbb1f5930b5f044df7edeef2f255ee.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org