[jira] [Created] (SPARK-35365) spark3.1.1 use too long to analyze table fields

["yao (Jira)" <ji...@apache.org>]

yao created SPARK-35365:
---------------------------

             Summary: spark3.1.1 use too long to analyze table fields
                 Key: SPARK-35365
                 URL: https://issues.apache.org/jira/browse/SPARK-35365
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.1.1
            Reporter: yao


I have a big sql with a few width tables join and complex logic, when I run that in spark 2.4 , it will take 20 minues in analyze phase, when I use spark 3.1.1, it will use about 40 minutes,

I need set spark.sql.analyzer.maxIterations=1000 in spark3.1.1.

or spark.sql.optimizer.maxIterations=1000 in spark2.4.

no other special setting for this .

I check on the spark ui , I find that there is no job generated, all executor have no active tasks, and when I set log level to debug, I find that the job is in analyze phase, analyze the fields reference.

this phase use too long time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org