[spark] branch branch-2.4 updated: [SPARK-33123][INFRA] Ignore GitHub only changes in Amplab Jenkins build

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new 15dc67e  [SPARK-33123][INFRA] Ignore GitHub only changes in Amplab Jenkins build
15dc67e is described below

commit 15dc67eb10339792bbf03a4717eefe0d95acb39b
Author: William Hyun <wi...@gmail.com>
AuthorDate: Mon Oct 19 14:13:37 2020 +0900

    [SPARK-33123][INFRA] Ignore GitHub only changes in Amplab Jenkins build
    
    ### What changes were proposed in this pull request?
    
    This PR aims to ignore GitHub only changes in Amplab Jenkins build.
    
    ### Why are the changes needed?
    
    This will save server resources.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is a dev-only change.
    
    ### How was this patch tested?
    
    Manually. I used the following doctest during testing and removed it at the clean-up.
    
    E2E tests:
    
    ```
    cd dev
    cat test.py
    ```
    
    ```python
    import importlib
    runtests = importlib.import_module("run-tests")
    print([x.name for x in runtests.determine_modules_for_files([".github/workflows/build_and_test.yml"])])
    ```
    
    ```python
    $ GITHUB_ACTIONS=1 python test.py
    ['root']
    $ python test.py
    []
    ```
    
    Unittests:
    
    ```bash
    $ GITHUN_ACTIONS=1 python3 -m doctest dev/run-tests.py
    $ python3 -m doctest dev/run-tests.py
    ```
    
    Closes #30020 from williamhyun/SPARK-33123.
    
    Lead-authored-by: William Hyun <wi...@gmail.com>
    Co-authored-by: Hyukjin Kwon <gu...@gmail.com>
    Signed-off-by: HyukjinKwon <gu...@apache.org>
    (cherry picked from commit e6c53c2c1b538d6272df4d1ca294d04f8b49bd6c)
    Signed-off-by: HyukjinKwon <gu...@apache.org>
---
 dev/run-tests.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/dev/run-tests.py b/dev/run-tests.py
index 58acffa..b726b0c 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -42,7 +42,8 @@ def determine_modules_for_files(filenames):
     """
     Given a list of filenames, return the set of modules that contain those files.
     If a file is not associated with a more specific submodule, then this method will consider that
-    file to belong to the 'root' module. GitHub Action and Appveyor files are ignored.
+    file to belong to the 'root' module. `.github` directory is counted only in GitHub Actions,
+    and `appveyor.yml` is always ignored because this file is dedicated only to AppVeyor builds.
 
     >>> sorted(x.name for x in determine_modules_for_files(["python/pyspark/a.py", "sql/core/foo"]))
     ['pyspark-core', 'sql']
@@ -55,6 +56,8 @@ def determine_modules_for_files(filenames):
     for filename in filenames:
         if filename in ("appveyor.yml",):
             continue
+        if ("GITHUB_ACTIONS" not in os.environ) and filename.startswith(".github"):
+            continue
         matched_at_least_one_module = False
         for module in modules.all_modules:
             if module.contains_file(filename):


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org