[GitHub] [spark] peter-toth commented on pull request #40744: [SPARK-24497][SQL] Support recursive SQL

["peter-toth (via GitHub)" <gi...@apache.org>]

peter-toth commented on PR #40744:
URL: https://github.com/apache/spark/pull/40744#issuecomment-1568607994

   > Thanks @peter-toth. I tested this patch locally. But it seem it throws `StackOverflowError`. How to reproduce:
   > 
   > ```
   > ./dev/make-distribution.sh --tgz  -Phive -Phive-thriftserver
   > tar -zxf spark-3.5.0-SNAPSHOT-bin-3.3.5.tgz
   > cd spark-3.5.0-SNAPSHOT-bin-3.3.5
   > bin/spark-sql
   > ```
   > 
   > ```
   > spark-sql (default)> WITH RECURSIVE t(n) AS (
   >                    >     VALUES (1)
   >                    > UNION ALL
   >                    >     SELECT n+1 FROM t WHERE n < 100
   >                    > )
   >                    > SELECT sum(n) FROM t;
   > 23/05/30 13:21:21 ERROR Executor: Exception in task 0.0 in stage 265.0 (TID 199)
   > java.lang.StackOverflowError
   > 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   > ```
   
   Thanks for testing this PR @wangyum. Iterestingly, I didn't encounter stack overflow when recursion level is <100. The error starts to appear at level ~170 in my tests. I guess this depends on your default stack size. Since recursion works in a way that each iteration depends on the previous iteration the RDD lineage of the tasks are getting bigger and bigger and the deserialization of those tasks can throw stack overflow error at some point. Let me amend this PR with adding optional checkpointing so as to truncate RDD linage and be able to deal with deeper recursion...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org