You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/07/28 09:19:25 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
LuciferYang opened a new pull request, #42203:
URL: https://github.com/apache/spark/pull/42203
### What changes were proposed in this pull request?
This pr aims add a new Daily testing GitHub Action job for Maven.
### Why are the changes needed?
Need Daily testing for Maven.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278695206
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
Review Comment:
will we also tests other java versions?
should we also mention the java version in the workflow name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278697204
##########
.github/workflows/build_maven.yml:
##########
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build using Maven (master, Scala 2.12, Hadoop 3, JDK 8)"
Review Comment:
mention the java version in the workflow name.Let's test Java 8 first, if necessary, we can add testing for Java 11 and Java 17 later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278703895
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
+ if [[ "$INCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test
+ elif [[ "$EXCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test
+ elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl connector/connect/client/jvm,connector/connect/common,connector/connect/server test
+ elif [[ "$MODULES_TO_TEST" == "sql#hive-thriftserver" ]]; then
+ # To avoid a compilation loop, for the `sql/hive-thriftserver` module, run `clean install` instead
+ ./build/mvn $MAVEN_CLI_OPTS -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Djava.version=${JAVA_VERSION/-ea} clean install
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} test
+ fi
Review Comment:
https://github.com/apache/spark/blob/b8cb9d552aaaa6d6d8211555d3d00fff1e394015/dev/run-tests.py#L373-L376
@zhengruifeng and It seems that `run_scala_tests_maven` currently only supports full module testing and does not support specified modules. Can we refactor `./dev/run-tests` in another pr, then re-use it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1657391699
cc @HyukjinKwon @zhengruifeng @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1698502467
Finally！ Maven Daily Tests passed.
https://github.com/apache/spark/actions/runs/6012638993
<img width="1011" alt="image" src="https://github.com/apache/spark/assets/1475305/05a40958-8766-41e0-9e5b-f19bd511a9aa">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278716506
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
Review Comment:
can use directly use `connector/kafka-0-10` in above `modules`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278715697
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
+ if [[ "$INCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test
+ elif [[ "$EXCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test
+ elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl connector/connect/client/jvm,connector/connect/common,connector/connect/server test
+ elif [[ "$MODULES_TO_TEST" == "sql#hive-thriftserver" ]]; then
+ # To avoid a compilation loop, for the `sql/hive-thriftserver` module, run `clean install` instead
+ ./build/mvn $MAVEN_CLI_OPTS -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Djava.version=${JAVA_VERSION/-ea} clean install
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} test
+ fi
Review Comment:
If `./dev/run-tests` can not be simply reused, I am fine with this change too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278702054
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
+ if [[ "$INCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test
+ elif [[ "$EXCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test
+ elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl connector/connect/client/jvm,connector/connect/common,connector/connect/server test
+ elif [[ "$MODULES_TO_TEST" == "sql#hive-thriftserver" ]]; then
+ # To avoid a compilation loop, for the `sql/hive-thriftserver` module, run `clean install` instead
+ ./build/mvn $MAVEN_CLI_OPTS -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Djava.version=${JAVA_VERSION/-ea} clean install
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} test
+ fi
Review Comment:
Have we used `build_spark_maven ` and `run_scala_tests_maven ` anywhere before? Are we sure it is available? I need some time to investigate this.
In addition, there are some compromises in the current build&test process, such as to avoid the `thriftserver` module falling into a dead loop during testing, only `clean install` can be executed for test now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278698220
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
+ if [[ "$INCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test
+ elif [[ "$EXCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test
+ elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl connector/connect/client/jvm,connector/connect/common,connector/connect/server test
+ elif [[ "$MODULES_TO_TEST" == "sql#hive-thriftserver" ]]; then
+ # To avoid a compilation loop, for the `sql/hive-thriftserver` module, run `clean install` instead
+ ./build/mvn $MAVEN_CLI_OPTS -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Djava.version=${JAVA_VERSION/-ea} clean install
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} test
+ fi
Review Comment:
this part is different from `build_and_test`,
shall we reuse `./dev/run-tests` with some configurations?
since I see methods `build_spark_maven` and `run_scala_tests_maven` in `run-tests.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278731066
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
Review Comment:
just to confirm, do we need `-Pconnect` for connect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1698512480
@LuciferYang thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1660108325
Thanks @zhengruifeng , let's monitor GA later :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278697204
##########
.github/workflows/build_maven.yml:
##########
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build using Maven (master, Scala 2.12, Hadoop 3, JDK 8)"
Review Comment:
already mention the java version in the workflow name.Let's test Java 8 first, if necessary, we can add testing for Java 11 and Java 17 later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278732844
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
Review Comment:
At least not now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1659569493
All test passed, could we merge this first and observe the GA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1657390926
https://github.com/LuciferYang/spark/actions/runs/5694334367/job/15435297051
The new tasks can be run, there are 3 module test failed：repl, connect, hive-thriftserver, I will create new jira to tracking them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278731939
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
Review Comment:
I tested this before, If `connector/kafka-0-10` is used, an error will be reported when `Upload test results to report` or `Upload unit tests log files` create the report file name, as filenames cannot have`/` (test-results-${{ matrix.modules }})
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278715697
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
+ export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} clean install
+ if [[ "$INCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.include.tags="$INCLUDED_TAGS" test
+ elif [[ "$EXCLUDED_TAGS" != "" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Djava.version=${JAVA_VERSION/-ea} -Dtest.exclude.tags="$EXCLUDED_TAGS" test
+ elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
+ ./build/mvn $MAVEN_CLI_OPTS -Djava.version=${JAVA_VERSION/-ea} -pl connector/connect/client/jvm,connector/connect/common,connector/connect/server test
+ elif [[ "$MODULES_TO_TEST" == "sql#hive-thriftserver" ]]; then
+ # To avoid a compilation loop, for the `sql/hive-thriftserver` module, run `clean install` instead
+ ./build/mvn $MAVEN_CLI_OPTS -pl sql/hive-thriftserver -Phive -Phive-thriftserver -Djava.version=${JAVA_VERSION/-ea} clean install
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn -Pmesos -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Pspark-ganglia-lgpl -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} test
+ fi
Review Comment:
If `./dev/run-tests` can not be simply reused, I am fine with directly using `maven` commands too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #42203:
URL: https://github.com/apache/spark/pull/42203#discussion_r1278731939
##########
.github/workflows/maven_test.yml:
##########
@@ -0,0 +1,210 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Build and test using Maven
+
+on:
+ workflow_call:
+ inputs:
+ java:
+ required: false
+ type: string
+ default: 8
+ branch:
+ description: Branch to run the build against
+ required: false
+ type: string
+ default: master
+ hadoop:
+ description: Hadoop version to run with. HADOOP_PROFILE environment variable should accept it.
+ required: false
+ type: string
+ default: hadoop3
+ envs:
+ description: Additional environment variables to set when running the tests. Should be in JSON format.
+ required: false
+ type: string
+ default: '{}'
+jobs:
+ # Build: build Spark and run the tests for specified modules using maven
+ build:
+ name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment }}"
+ runs-on: ubuntu-22.04
+ strategy:
+ fail-fast: false
+ matrix:
+ java:
+ - ${{ inputs.java }}
+ hadoop:
+ - ${{ inputs.hadoop }}
+ hive:
+ - hive2.3
+ modules:
+ - >-
+ core,repl,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+ - >-
+ graphx,streaming,mllib-local,mllib,hadoop-cloud
+ - >-
+ sql#hive-thriftserver
+ - >-
+ connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
+ - >-
+ sql#catalyst,resource-managers#yarn,resource-managers#mesos,resource-managers#kubernetes#core
+ - >-
+ connect
+ # Here, we split Hive and SQL tests into some of slow ones and the rest of them.
+ included-tags: [ "" ]
+ excluded-tags: [ "" ]
+ comment: [ "" ]
+ include:
+ # Hive tests
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- slow tests"
+ - modules: sql#hive
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.SlowHiveTest
+ comment: "- other tests"
+ # SQL tests
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.ExtendedSQLTest
+ comment: "- extended tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ included-tags: org.apache.spark.tags.SlowSQLTest
+ comment: "- slow tests"
+ - modules: sql#core
+ java: ${{ inputs.java }}
+ hadoop: ${{ inputs.hadoop }}
+ hive: hive2.3
+ excluded-tags: org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
+ comment: "- other tests"
+ env:
+ MODULES_TO_TEST: ${{ matrix.modules }}
+ EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
+ INCLUDED_TAGS: ${{ matrix.included-tags }}
+ HADOOP_PROFILE: ${{ matrix.hadoop }}
+ HIVE_PROFILE: ${{ matrix.hive }}
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v3
+ # In order to fetch changed files
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+ # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v3
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v3
+ with:
+ path: ~/.cache/coursier
+ key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ - name: Install Java ${{ matrix.java }}
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: ${{ matrix.java }}
+ - name: Install Python 3.8
+ uses: actions/setup-python@v4
+ # We should install one Python that is higher than 3+ for SQL and Yarn because:
+ # - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
+ # - Yarn has a Python specific test too, for example, YarnClusterSuite.
+ if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core'))
+ with:
+ python-version: 3.8
+ architecture: x64
+ - name: Install Python packages (Python 3.8)
+ if: (contains(matrix.modules, 'sql#core'))
+ run: |
+ python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.20.3'
+ python3.8 -m pip list
+ # Run the tests.
+ - name: Run tests
+ env: ${{ fromJSON(inputs.envs) }}
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ # Fix for TTY related issues when launching the Ammonite REPL in tests.
+ export TERM=vt100 && script -qfc 'echo exit | amm -s' && rm typescript
+ # `set -e` to make the exit status as expected due to use `script -q -e -c` to run the commands
+ set -e
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g -XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ matrix.java }}
+ # Replace with the real module name, for example, connector#kafka-0-10 -> connector/kafka-0-10
Review Comment:
I tested this before, If `connector/kafka-0-10` is used, an error will be reported when `Upload test results to report` or `Upload unit tests log files` create the report file name, as filenames cannot have`/` (${{ matrix.modules }})
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
URL: https://github.com/apache/spark/pull/42203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42203: [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42203:
URL: https://github.com/apache/spark/pull/42203#issuecomment-1660024239
merged to master, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org