[jira] [Created] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

["Danny Guinther (Jira)" <ji...@apache.org>]

Danny Guinther created SPARK-38792:
--------------------------------------

             Summary: Regression in time executor takes to do work since v3.0.1 ?
                 Key: SPARK-38792
                 URL: https://issues.apache.org/jira/browse/SPARK-38792
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.2.1
            Reporter: Danny Guinther
         Attachments: what-s-up-with-exec-actions.jpg

Hello!

I'm sorry to trouble you with this, but I'm seeing a noticeable regression in performance when upgrading from 3.0.1 to 3.2.1 and I can't pin down why. I don't believe it is specific to my application since the upgrade to 3.0.1 to 3.2.1 is purely a configuration change. I'd guess it presents itself in my application due to the high volume of work my application does, but I could be mistaken.

The gist is that it seems like the executor actions I'm running suddenly appear to take a lot longer on Spark 3.2.1. I don't have any ability to test versions between 3.0.1 and 3.2.1 because my application was previously blocked from upgrading beyond Spark 3.0.1 by https://issues.apache.org/jira/browse/SPARK-37391 (which I helped to fix).

Any ideas what might cause this or metrics I might try to gather to pinpoint the problem? I've tried a bunch of the suggestions from [https://spark.apache.org/docs/latest/tuning.html] to see if any of those help, but none of the adjustments I've tried have been fruitful. I also tried to look in [https://spark.apache.org/docs/latest/sql-migration-guide.html] for ideas as to what might have changed to cause this behavior, but haven't seen anything that sticks out as being a possible source of the problem.

I have attached a graph that shows the drastic change in time taken by executor actions. In the image the blue and purple lines are different kinds of reads using the built-in JDBC data reader and the green line is writes using a custom-built data writer. The deploy to switch from 3.0.1 to 3.2.1 occurred at 9AM on the graph.

 

Thanks in advance for any help!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org