You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by ka...@apache.org on 2021/07/11 14:46:09 UTC
[submarine] branch master updated: SUBMARINE-895. Refactor GitHub
Actions YAML files
This is an automated email from the ASF dual-hosted git repository.
kaihsun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 8dec01f SUBMARINE-895. Refactor GitHub Actions YAML files
8dec01f is described below
commit 8dec01f9aef9098223c49aea691663480b69a716
Author: KUAN-HSUN-LI <b0...@ntu.edu.tw>
AuthorDate: Thu Jul 8 17:59:43 2021 +0800
SUBMARINE-895. Refactor GitHub Actions YAML files
### What is this PR for?
* Some GitHub Actions' YAML files are writing with different code styles and using some old version packages. Improve the code styles and update the packages.
* Add the Licence on the top of files
### What type of PR is it?
[Refactoring]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-895
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: KUAN-HSUN-LI <b0...@ntu.edu.tw>
Signed-off-by: Kai-Hsun Chen <ka...@apache.org>
Closes #657 from KUAN-HSUN-LI/SUBMARINE-895 and squashes the following commits:
b3d6ee98 [KUAN-HSUN-LI] fix path error
cbc7aff0 [KUAN-HSUN-LI] SUBMARINE-895. Refactor GitHub Actions' YAML files
---
.github/workflows/deploy_docker_images.yml | 21 ++++--
.github/workflows/deploy_website.yml | 28 ++++++--
.github/workflows/master.yml | 62 +++++++++--------
.github/workflows/python.yml | 108 +++++++++--------------------
4 files changed, 105 insertions(+), 114 deletions(-)
diff --git a/.github/workflows/deploy_docker_images.yml b/.github/workflows/deploy_docker_images.yml
index 8a45b49..3afd32e 100644
--- a/.github/workflows/deploy_docker_images.yml
+++ b/.github/workflows/deploy_docker_images.yml
@@ -1,5 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
name: Deploy submarine docker images
+# Trigger the workflow on master branch push
on:
push:
branches: [master]
@@ -8,13 +24,10 @@ jobs:
if: github.repository == 'apache/submarine'
runs-on: ubuntu-latest
timeout-minutes: 240
- strategy:
- fail-fast: true
env:
SUBMARINE_VERSION: 0.6.0-SNAPSHOT
steps:
- - name: Check out code
- uses: actions/checkout@v2
+ - uses: actions/checkout@v2
with:
repository: apache/submarine
- uses: docker/login-action@v1
diff --git a/.github/workflows/deploy_website.yml b/.github/workflows/deploy_website.yml
index 82ab9ba..fc994ae 100644
--- a/.github/workflows/deploy_website.yml
+++ b/.github/workflows/deploy_website.yml
@@ -1,5 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
name: Deploy Submarine documentation
+# Trigger the workflow on master branch push and pull request
on:
pull_request:
branches: [master]
@@ -11,13 +27,13 @@ jobs:
if: github.event_name != 'push'
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v1
- - uses: actions/setup-node@v1
+ - uses: actions/checkout@v2
+ - uses: actions/setup-node@v2
with:
node-version: '12.x'
- name: Test Build
+ working-directory: ./website
run: |
- cd website
if [ -e yarn.lock ]; then
yarn install --frozen-lockfile
elif [ -e package-lock.json ]; then
@@ -30,8 +46,8 @@ jobs:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v1
- - uses: actions/setup-node@v1
+ - uses: actions/checkout@v2
+ - uses: actions/setup-node@v2
with:
node-version: '12.x'
- name: Add key to allow access to repository
@@ -47,8 +63,8 @@ jobs:
USE_SSH: true
GIT_USER: pingsutw
DEPLOYMENT_BRANCH: asf-site
+ working-directory: ./website
run: |
- cd website
git config --global user.email "pingsutw@apache.org"
git config --global user.name "pingsutw"
if [ -e yarn.lock ]; then
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 939d0cc..8e7435c 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -1,5 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
name: Submarine
+# Trigger the workflow on push or pull request
on: [push, pull_request]
env:
@@ -22,8 +38,8 @@ env:
jobs:
build:
- name: Build
runs-on: ubuntu-latest
+ timeout-minutes: 30
steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
@@ -57,6 +73,7 @@ jobs:
path: submarine-dist/target/submarine-dist*.tar.gz
submarine-e2e:
runs-on: ubuntu-latest
+ timeout-minutes: 30
needs: [build]
services:
mysql:
@@ -108,6 +125,7 @@ jobs:
mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B
submarine-k8s:
runs-on: ubuntu-latest
+ timeout-minutes: 30
needs: [build]
steps:
- uses: actions/checkout@v2
@@ -134,15 +152,17 @@ jobs:
helm version
kind version
- name: Create kind cluster
- run: |
- kind create cluster --config ./.github/config/kind-config-kind.yaml --wait 3m --image kindest/node:v1.15.12
+ run: kind create cluster --config ./.github/config/kind-config-kind.yaml --wait 3m --image kindest/node:v1.15.12
- name: Show K8s cluster information
run: |
kubectl cluster-info
kubectl version
kubectl get pods -n kube-system
+
+ # test-k8s require this config
export KUBECONFIG=~/.kube/kind-config-kind
kind export kubeconfig --kubeconfig ${KUBECONFIG}
+
echo "current-context:" $(kubectl config current-context)
echo "environment-kubeconfig:" ${KUBECONFIG}
- name: Download compiled Submarine binaries
@@ -156,8 +176,7 @@ jobs:
mv submarine-dist*.tar.gz submarine-dist/target/
sudo chmod -R a+rwX submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
- name: Build Image locally
- run: |
- .github/scripts/build-image-locally.sh
+ run: .github/scripts/build-image-locally.sh
- name: Start submarine
run: bash ./.github/scripts/start-submarine.sh
- name: Test
@@ -170,9 +189,9 @@ jobs:
mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B
- name: Failure status
run: |
+ kubectl describe nodes
kubectl get pods
kubectl -n default get events --sort-by='{.lastTimestamp}'
- kubectl describe nodes
if: ${{ failure() }}
submarine-commons-cluster:
runs-on: ubuntu-latest
@@ -353,6 +372,7 @@ jobs:
MYSQL_ROOT_PASSWORD: "password"
ports:
- 3306:3306
+ # wait until mysql is health
options: --health-cmd "mysqladmin ping" --health-interval 10s --health-timeout 10s --health-retries 10
steps:
- uses: actions/checkout@v2
@@ -415,8 +435,7 @@ jobs:
mvn $BUILD_FLAG $MODULES -B
- name: NPM install
working-directory: ./submarine-workbench/workbench-web
- run: |
- npm install
+ run: npm install
- name: Test with chrome
working-directory: ./submarine-workbench/workbench-web
run: npm run test -- --no-watch --no-progress --browsers=ChromeHeadlessCI
@@ -507,6 +526,7 @@ jobs:
linter:
name: Check Style
runs-on: ubuntu-latest
+ timeout-minutes: 30
steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
@@ -521,30 +541,12 @@ jobs:
run: |
mvn --version
java -version
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/com
- key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-com-
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/org
- key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-org-
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/net
- key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-net-
- - uses: actions/cache@v1
+ - uses: actions/cache@v2
with:
- path: ~/.m2/repository/io
- key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
+ path: ~/.m2/repository
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ runner.os }}-maven-io-
+ ${{ runner.os }}-maven-
- name: Java Style
run: ./dev-support/style-check/lint-java.sh
- name: Scala Style
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index 70617ea..fbc79e2 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -1,3 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
name: python-sdk
# Trigger the workflow on push or pull request
@@ -6,19 +21,16 @@ on: [push, pull_request]
jobs:
unit:
runs-on: ubuntu-latest
- timeout-minutes: 120
+ timeout-minutes: 30
strategy:
matrix:
python-version: [3.6, 3.7]
tf-version: [1.14.0, 1.15.0]
fail-fast: false
- env:
- PYTHON: ${{ matrix.python }}
steps:
- - name: Checkout project
- uses: actions/checkout@v1
+ - uses: actions/checkout@v2
- name: Setup python environment
- uses: actions/setup-python@v1
+ uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
@@ -30,16 +42,16 @@ jobs:
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
pip install -r ./submarine-sdk/pysubmarine/github-actions/lint-requirements.txt
- name: Check python sdk code style
- run: |
- ./submarine-sdk/pysubmarine/github-actions/lint.sh
+ run: ./submarine-sdk/pysubmarine/github-actions/lint.sh
- name: Run unit test
- run: |
- pytest --cov=submarine -vs -m "not e2e"
+ run: pytest --cov=submarine -vs -m "not e2e"
integration:
runs-on: ubuntu-latest
- timeout-minutes: 120
+ timeout-minutes: 60
steps:
- uses: actions/checkout@v2
+ with:
+ fetch-depth: 50
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
@@ -52,31 +64,8 @@ jobs:
run: |
mvn --version
java -version
- - name: Create the kind config
- run: |
- cat <<EOF > ./kind-config-kind.yaml
- kind: Cluster
- apiVersion: kind.x-k8s.io/v1alpha4
- nodes:
- - role: control-plane
- kubeadmConfigPatches:
- - |
- kind: InitConfiguration
- nodeRegistration:
- kubeletExtraArgs:
- node-labels: "ingress-ready=true"
- authorization-mode: "AlwaysAllow"
- extraPortMappings:
- - containerPort: 32080
- hostPort: 80
- protocol: TCP
- - containerPort: 443
- hostPort: 443
- protocol: TCP
- EOF
- name: Create kind cluster
- run: |
- kind create cluster --config ./kind-config-kind.yaml --wait 3m --image kindest/node:v1.15.12
+ run: kind create cluster --config ./.github/config/kind-config-kind.yaml --wait 3m --image kindest/node:v1.15.12
- name: Show K8s cluster information
run: |
kubectl cluster-info
@@ -85,46 +74,18 @@ jobs:
echo "current-context:" $(kubectl config current-context)
echo "environment-kubeconfig:" ${KUBECONFIG}
# Cache maven (for submarine server)
- - uses: actions/cache@v1
+ - uses: actions/cache@v2
with:
- path: ~/.m2/repository/com
- key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
+ path: ~/.m2/repository
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ runner.os }}-maven-com-
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/org
- key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-org-
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/net
- key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-net-
- - uses: actions/cache@v1
- with:
- path: ~/.m2/repository/io
- key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-io-
+ ${{ runner.os }}-maven-
- name: Build Image locally
- run: |
- .github/scripts/build-image-locally.sh
+ run: .github/scripts/build-image-locally.sh
- name: Start submarine
- run: |
- # Fix submarine-database start failed in kind. https://github.com/kubernetes/minikube/issues/7906
- sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
- sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
- helm install --wait submarine ./helm-charts/submarine
- kubectl get pods
- kubectl port-forward svc/submarine-database 3306:3306 &
- kubectl port-forward svc/submarine-server 8080:8080 &
- kubectl port-forward svc/submarine-minio-service 9000:9000 &
- kubectl port-forward svc/submarine-mlflow-service 5001:5000 &
+ run: bash ./.github/scripts/start-submarine.sh
- name: Setup python environment
- uses: actions/setup-python@v1
+ uses: actions/setup-python@v2
with:
python-version: 3.6
- name: Install dependencies
@@ -133,12 +94,11 @@ jobs:
pip install --no-cache-dir -e ./submarine-sdk/pysubmarine/.[tf,pytorch]
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- name: Run integration test
- run: |
- cd submarine-sdk/pysubmarine
- pytest --cov=submarine -vs -m "e2e"
+ working-directory: ./submarine-sdk/pysubmarine
+ run: pytest --cov=submarine -vs -m "e2e"
- name: Failure status
run: |
+ kubectl describe nodes
kubectl get pods
kubectl -n default get events --sort-by='{.lastTimestamp}'
- kubectl describe nodes
if: ${{ failure() }}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org