You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2023/07/24 21:29:00 UTC

[jira] [Created] (SPARK-44529) Add a flag to resolve docker tags to hashes at launch time

Holden Karau created SPARK-44529:
------------------------------------

             Summary: Add a flag to resolve docker tags to hashes at launch time
                 Key: SPARK-44529
                 URL: https://issues.apache.org/jira/browse/SPARK-44529
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 3.5.0, 4.0.0
            Reporter: Holden Karau


If you have an Spark docker tag (like say 3.3) you might want to update the container but only for newly launched jobs not existing jobs. To allow this we can resolve the tag to hash at launch time.

 

In some environments this may also offer a small performance improvement as it saves K8s from having to re-resolve the tag with additional executor launches.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org