You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/01/12 10:52:34 UTC
(spark) branch master updated: [SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9f061b0d03a3 [SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps
9f061b0d03a3 is described below
commit 9f061b0d03a3847f48329f70e05e2936aeac91c9
Author: panbingkun <pa...@baidu.com>
AuthorDate: Fri Jan 12 02:52:24 2024 -0800
[SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps
### What changes were proposed in this pull request?
The pr aims to fix potential environmental variable(eg: `PYTHON_TO_TEST`.) transfer issue in `pyspark job GA`
### Why are the changes needed?
https://github.com/apache/spark/actions/workflows/build_python.yml
Let's take a successful `build_ python` Using as an example, https://github.com/apache/spark/actions/runs/7476792796/job/20348090667
1.python 3.10
Obviously, it is meaningless to enumerate the packages of `Python 3.9` when building PySpark based on `Python 3.10`.
https://github.com/apache/spark/actions/runs/7476792796/job/20348079659
<img width="1351" alt="image" src="https://github.com/apache/spark/assets/15246973/7f8d1f80-f0ce-458c-968b-0c66ee456410">
2.python 3.11
https://github.com/apache/spark/actions/runs/7476792796/job/20348091081
<img width="1419" alt="image" src="https://github.com/apache/spark/assets/15246973/5bb77e2e-9317-4189-8a09-174c923d515b">
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44698 from panbingkun/SPARK-46692.
Authored-by: panbingkun <pa...@baidu.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.github/workflows/build_and_test.yml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index beeef93fee03..089a9296bcc4 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -430,6 +430,7 @@ jobs:
distribution: zulu
java-version: ${{ matrix.java }}
- name: List Python packages (${{ env.PYTHON_TO_TEST }})
+ env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
for py in $(echo $PYTHON_TO_TEST | tr "," "\n")
@@ -467,12 +468,14 @@ jobs:
flags: unittests
name: PySpark
- name: Upload test results to report
+ env: ${{ fromJSON(inputs.envs) }}
if: always()
uses: actions/upload-artifact@v3
with:
name: test-results-${{ matrix.modules }}--${{ matrix.java }}-${{ inputs.hadoop }}-hive2.3
path: "**/target/test-reports/*.xml"
- name: Upload unit tests log files
+ env: ${{ fromJSON(inputs.envs) }}
if: ${{ !success() }}
uses: actions/upload-artifact@v3
with:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org