[jira] [Created] (SPARK-39603) Dataset planning in a unit test takes a very long time to finish (e.g. >8mins for complex job)

["Tanin Na Nakorn (Jira)" <ji...@apache.org>]

Tanin Na Nakorn created SPARK-39603:
---------------------------------------

             Summary: Dataset planning in a unit test takes a very long time to finish (e.g. >8mins for complex job)
                 Key: SPARK-39603
                 URL: https://issues.apache.org/jira/browse/SPARK-39603
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.3.0
            Reporter: Tanin Na Nakorn


At Stripe, we have a very complex data job. The unit test was running fine when we used RDD.

After we switched to Dataset, the unit test takes considerably longer (e.g. > 8 mins just for planning).

Most of our unit tests only process 1-2 records.

We have tried to investigate it a bit, and we are somewhat sure it's the planning phrase.

We tried disabling almost all optimizers except the ~10 optimizers that can't be disabled. It doesn't impact the test run time at all.

Is there a way to make dataset plan faster in unit test.

Thank you!

(Please excuse us. I may use inaccurate term.)



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org