[GitHub] [spark] bersprockets commented on pull request #35233: [SPARK-37290][SQL] - Exponential planning time in case of non-deterministic function

[GitBox <gi...@apache.org>]

bersprockets commented on pull request #35233:
URL: https://github.com/apache/spark/pull/35233#issuecomment-1028471972


   @Stelyus 
   
   Maybe a more concise reproduction test would be:
   ```
   val df = spark.range(10).select((1 to 28).map(i => $"id".alias(s"col$i")): _*)
   df.selectExpr("rand() as col0", "*").createOrReplaceTempView("tbl1")
   sql("select l.* from tbl1 l join tbl1 r on l.col1 = r.col1").explain
   ```
   Or, if you use spark-sql:
   ```
   create or replace temp view tbl as
   select rand() c0, id c1, id c2, id c3, id c4, id c5, id c6, id c7, id c8, id c9, id c10, id c11,
     id c12, id c13, id c14, id c15, id c16, id c17, id c18, id c19, id c20, id c21, id c22, id c23,
     id c24, id c25, id c26, id c27, id c28
   from range(10);
   
   explain select l.* from tbl l join tbl r on l.c1 = r.c1;
   ```
   Both will blow out a 5G driver.
   
   This was introduced by #29598, which changed `allConstraints` in `getAllValidConstraints` from a `Set[Expression]` to an `ExpressionSet`. Before the change, the non-deterministic expression could be added only once. Now, as you already explained, the non-deterministic expression gets re-added (doubling in number for each item in projectList) because `ExpressionSet.originals` is an `ArrayBuffer`, not a `Set`.
   
   I wonder if `ExpressionSet.originals` could also be a `Set`?
   
   cc @hvanhovell @dbaliafroozeh 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org