[jira] [Commented] (SPARK-39522) Uses Docker image cache over a custom image

["Yikun Jiang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558067#comment-17558067 ] 

Yikun Jiang commented on SPARK-39522:
-------------------------------------

I have some questions here, the cache seems to just reduce the build time (docker build), and the current infra uses static sha images (docker run/pull), so in theory there won't be any improvement.

So, what you mean is that we want to migrate dongjoon-hyun/ApacheSparkGitHubActionImage to apache/spark, and then in each pyspark workflow, do a full docker build in every pr triggered job, and speed it up by caching?

> Uses Docker image cache over a custom image
> -------------------------------------------
>
>                 Key: SPARK-39522
>                 URL: https://issues.apache.org/jira/browse/SPARK-39522
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Project Infra
>    Affects Versions: 3.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>
> We should probably replace the base image (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain ubunto image w/ Docker image cache. See also https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org