[GitHub] [spark] Yikun commented on a diff in pull request #37828: [SPARK-40384][INFRA] Only do base image real in time build when infra dockerfile is changed

[GitBox <gi...@apache.org>]

Yikun commented on code in PR #37828:
URL: https://github.com/apache/spark/pull/37828#discussion_r969591362


##########
.github/workflows/build_and_test.yml:
##########
@@ -87,6 +91,7 @@ jobs:
             sparkr=`./dev/is-changed.py -m sparkr`
             tpcds=`./dev/is-changed.py -m sql`
             docker=`./dev/is-changed.py -m docker-integration-tests`
+            infra_image=`./dev/is-changed.py -m infra-image`

Review Comment:
   > would that still build the base image when there are other changes together with Dockerfile?
   
   Yes, `module_names` contains all changed module's name, test_modules will be ['infra-image'], this will return true if `infra-image` module is detected:
   
   ```python
       elif len(set(test_modules).intersection(module_names)) == 0:
           print("false")
           if opts.fail:
               sys.exit(1)
       else:
           print("true")
   ```
   
   BTW, If the fork repository is not synchronized to the latest, if the dockerfile changes are included in the synchronized code, it will also return true. You could see test https://github.com/Yikun/spark/pull/170 , [infra-image return true](https://github.com/Yikun/spark/actions/runs/3012667538/jobs/4840989926#step:4:45), then [build](https://github.com/Yikun/spark/actions/runs/3012667538/jobs/4841457546#step:7:102) and use the base image in [lint](https://github.com/Yikun/spark/actions/runs/3012667538/jobs/4841486118#step:2:19), [sparkr](https://github.com/Yikun/spark/actions/runs/3012667538/jobs/4841485726#step:2:19) and [pyspark](https://github.com/Yikun/spark/actions/runs/3012667538/jobs/4841485816#step:2:19).
   
   > Do we return true when root module is detected?
   
   No, we only return true when infra dockerfile changes.
   
   If only root module is detected, will first skip this root check
   
   ```python
       # `./dev/is-changed.py -m infra-image` == True only when changing the infra dockerfile
       elif ("root" in test_modules or modules.root in changed_modules) and (
           ["infra-image"] != test_modules
       ):
   ```
   
   and then do continue check as above return result according `infra-image` module is detected. The current PR is an real example to show this case, root dectected (because we change the yaml), but didn't change the dockerfile, [infra-image return false](https://github.com/Yikun/spark/runs/8242270891#step:4:45), then use the [`master-static` image](https://github.com/Yikun/spark/actions/runs/3012600557/jobs/4840983854#step:2:20).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org