You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/10/28 02:01:07 UTC
(spark) branch master updated: [SPARK-50100][INFRA] Remove `branch-3.4`-related CIs and comments
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d2ffccbe9cd0 [SPARK-50100][INFRA] Remove `branch-3.4`-related CIs and comments
d2ffccbe9cd0 is described below
commit d2ffccbe9cd014ebdb4fcb2710e3ea67d2a3791f
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Oct 27 19:00:49 2024 -0700
[SPARK-50100][INFRA] Remove `branch-3.4`-related CIs and comments
### What changes were proposed in this pull request?
This PR aims to remove `branch-3.4`-related CIs and comments.
- https://github.com/apache/spark/actions/workflows/build_branch34.yml
- https://github.com/apache/spark/actions/workflows/build_branch34_python.yml
### Why are the changes needed?
Apache Spark 3.4.4 is released Today and `branch-3.4` reaches the end of support. By removing `branch-3.4` related stuff, we can
- Minimize `GitHub Action` CI usage to maintain it under ASF policy.
- Prepare the room to set up `branch-4.0` branch for Apache Spark 4.0.0 on January 2025.
- Simplify the INFRA code
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
**BEFORE**
```
$ git grep branch-3.4 | wc -l
20
```
**AFTER**
```
$ git grep branch-3.4 | wc -l
0
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48670 from dongjoon-hyun/SPARK-50100.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.github/workflows/build_and_test.yml | 25 +++++---------
.github/workflows/build_branch34.yml | 51 -----------------------------
.github/workflows/build_branch34_python.yml | 45 -------------------------
.github/workflows/publish_snapshot.yml | 8 ++---
dev/protobuf-breaking-changes-check.sh | 2 +-
5 files changed, 14 insertions(+), 117 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 8143a9857a67..e51c896cc3f9 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -702,13 +702,6 @@ jobs:
run: ./dev/lint-java
- name: Spark connect jvm client mima check
run: ./dev/connect-jvm-client-mima-check
- - name: Install Python linter dependencies for branch-3.4
- if: inputs.branch == 'branch-3.4'
- run: |
- # SPARK-44554: Copy from https://github.com/apache/spark/blob/a05c27e85829fe742c1828507a1fd180cdc84b54/.github/workflows/build_and_test.yml#L571-L578
- # Should delete this section after SPARK 3.4 EOL.
- python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.920' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc 'jinja2<3.0.0' 'black==22.6.0'
- python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython 'grpcio==1.48.1' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0'
- name: Install Python linter dependencies for branch-3.5
if: inputs.branch == 'branch-3.5'
run: |
@@ -717,7 +710,7 @@ jobs:
python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc 'jinja2<3.0.0' 'black==22.6.0'
python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython 'grpcio==1.56.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0'
- name: Install Python dependencies for python linter and documentation generation
- if: inputs.branch != 'branch-3.4' && inputs.branch != 'branch-3.5'
+ if: inputs.branch != 'branch-3.5'
run: |
# Should unpin 'sphinxcontrib-*' after upgrading sphinx>5
# See 'ipython_genutils' in SPARK-38517
@@ -745,16 +738,16 @@ jobs:
if: inputs.branch == 'branch-3.5'
run: if test -f ./dev/connect-check-protos.py; then PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9 ./dev/connect-check-protos.py; fi
# Should delete this section after SPARK 3.5 EOL.
- - name: Install JavaScript linter dependencies for branch-3.4, branch-3.5
- if: inputs.branch == 'branch-3.4' || inputs.branch == 'branch-3.5'
+ - name: Install JavaScript linter dependencies for branch-3.5
+ if: inputs.branch == 'branch-3.5'
run: |
apt update
apt-get install -y nodejs npm
- name: JS linter
run: ./dev/lint-js
# Should delete this section after SPARK 3.5 EOL.
- - name: Install R linter dependencies for branch-3.4, branch-3.5
- if: inputs.branch == 'branch-3.4' || inputs.branch == 'branch-3.5'
+ - name: Install R linter dependencies for branch-3.5
+ if: inputs.branch == 'branch-3.5'
run: |
apt update
apt-get install -y libcurl4-openssl-dev libgit2-dev libssl-dev libxml2-dev \
@@ -834,7 +827,7 @@ jobs:
distribution: zulu
java-version: ${{ inputs.java }}
- name: Install Python dependencies for python linter and documentation generation
- if: inputs.branch != 'branch-3.4' && inputs.branch != 'branch-3.5'
+ if: inputs.branch != 'branch-3.5'
run: |
# Should unpin 'sphinxcontrib-*' after upgrading sphinx>5
# See 'ipython_genutils' in SPARK-38517
@@ -845,8 +838,8 @@ jobs:
'pandas-stubs==1.2.0.53' 'grpcio==1.62.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
'sphinxcontrib-applehelp==1.0.4' 'sphinxcontrib-devhelp==1.0.2' 'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' 'sphinxcontrib-serializinghtml==1.1.5'
python3.9 -m pip list
- - name: Install dependencies for documentation generation for branch-3.4, branch-3.5
- if: inputs.branch == 'branch-3.4' || inputs.branch == 'branch-3.5'
+ - name: Install dependencies for documentation generation for branch-3.5
+ if: inputs.branch == 'branch-3.5'
run: |
# pandoc is required to generate PySpark APIs as well in nbsphinx.
apt-get update -y
@@ -1134,7 +1127,7 @@ jobs:
export PVC_TESTS_VM_PATH=$PVC_TMP_DIR
minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH} --gid=0 --uid=185 &
kubectl create clusterrolebinding serviceaccounts-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts || true
- if [[ "${{ inputs.branch }}" == 'branch-3.5' || "${{ inputs.branch }}" == 'branch-3.4' ]]; then
+ if [[ "${{ inputs.branch }}" == 'branch-3.5' ]]; then
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.7.0/installer/volcano-development.yaml || true
else
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.9.0/installer/volcano-development.yaml || true
diff --git a/.github/workflows/build_branch34.yml b/.github/workflows/build_branch34.yml
deleted file mode 100644
index deb6c4240797..000000000000
--- a/.github/workflows/build_branch34.yml
+++ /dev/null
@@ -1,51 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build (branch-3.4, Scala 2.13, Hadoop 3, JDK 8)"
-
-on:
- schedule:
- - cron: '0 9 * * *'
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 8
- branch: branch-3.4
- hadoop: hadoop3
- envs: >-
- {
- "SCALA_PROFILE": "scala2.13",
- "PYTHON_TO_TEST": "",
- "ORACLE_DOCKER_IMAGE_NAME": "gvenzl/oracle-xe:21.3.0"
- }
- jobs: >-
- {
- "build": "true",
- "sparkr": "true",
- "tpcds-1g": "true",
- "docker-integration-tests": "true",
- "k8s-integration-tests": "true",
- "lint" : "true"
- }
diff --git a/.github/workflows/build_branch34_python.yml b/.github/workflows/build_branch34_python.yml
deleted file mode 100644
index c109ba2dc792..000000000000
--- a/.github/workflows/build_branch34_python.yml
+++ /dev/null
@@ -1,45 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Python-only (branch-3.4)"
-
-on:
- schedule:
- - cron: '0 9 * * *'
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 8
- branch: branch-3.4
- hadoop: hadoop3
- envs: >-
- {
- "PYTHON_TO_TEST": ""
- }
- jobs: >-
- {
- "pyspark": "true",
- "pyspark-pandas": "true"
- }
diff --git a/.github/workflows/publish_snapshot.yml b/.github/workflows/publish_snapshot.yml
index 1b5bd0ba6128..a5854d96a4d1 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish_snapshot.yml
@@ -28,7 +28,7 @@ on:
description: 'list of branches to publish (JSON)'
required: true
# keep in sync with default value of strategy matrix 'branch'
- default: '["master", "branch-3.5", "branch-3.4"]'
+ default: '["master", "branch-3.5"]'
jobs:
publish-snapshot:
@@ -38,7 +38,7 @@ jobs:
fail-fast: false
matrix:
# keep in sync with default value of workflow_dispatch input 'branch'
- branch: ${{ fromJSON( inputs.branch || '["master", "branch-3.5", "branch-3.4"]' ) }}
+ branch: ${{ fromJSON( inputs.branch || '["master", "branch-3.5"]' ) }}
steps:
- name: Checkout Spark repository
uses: actions/checkout@v4
@@ -52,13 +52,13 @@ jobs:
restore-keys: |
snapshot-maven-
- name: Install Java 8 for branch-3.x
- if: matrix.branch == 'branch-3.5' || matrix.branch == 'branch-3.4'
+ if: matrix.branch == 'branch-3.5'
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 8
- name: Install Java 17
- if: matrix.branch != 'branch-3.5' && matrix.branch != 'branch-3.4'
+ if: matrix.branch != 'branch-3.5'
uses: actions/setup-java@v4
with:
distribution: temurin
diff --git a/dev/protobuf-breaking-changes-check.sh b/dev/protobuf-breaking-changes-check.sh
index 05d90be564ab..327e54be63e6 100755
--- a/dev/protobuf-breaking-changes-check.sh
+++ b/dev/protobuf-breaking-changes-check.sh
@@ -21,7 +21,7 @@ set -ex
if [[ $# -gt 1 ]]; then
echo "Illegal number of parameters."
echo "Usage: ./dev/protobuf-breaking-changes-check.sh [branch]"
- echo "the default branch is 'master', available options are 'master', 'branch-3.4', etc"
+ echo "the default branch is 'master'"
exit -1
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org