Status of Spark testing on ARM64

[Tianhua huang <hu...@gmail.com>]

Hi all,
I will give you some informations about ARM CI of Spark:

Our team and community are working on build/test Spark master on ARM64
server, after find and fix some issues[1], we have integrated two ARM
testing jobs[2] to community CI(AMPLAB Jenkins),
they run as daily job and have been stablely running for few weeks, and the
two ARM testing jobs are success generally.
Thanks Sean Owen, Shane Knapp, Dongjoon Hyun and community to help us :)

If you are interested, please have a try:)  Before
https://github.com/apache/spark/pull/26636 merged, you have to download and
maven install org.openlabtesting.leveldbjni:leveldbjni-all.1.8 using
commands:
wget
https://repo1.maven.org/maven2/org/openlabtesting/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar

mvn install:install-file -DgroupId=org.fusesource.leveldbjni
-DartifactId=leveldbjni-all -Dversion=1.8 -Dpackaging=jar
-Dfile=leveldbjni-all-1.8.jar
Then, you can build and test Spark on ARM64 server.

If you have any questions, please don't hesitate to contact me, thanks all!

[1]:
https://issues.apache.org/jira/browse/SPARK-28770 （
https://github.com/apache/spark/pull/25673）
https://issues.apache.org/jira/browse/SPARK-28519 （
https://github.com/apache/spark/pull/25279）
https://issues.apache.org/jira/browse/SPARK-28433 （
https://github.com/apache/spark/pull/25186）
https://issues.apache.org/jira/browse/SPARK-28467 （
https://github.com/apache/spark/pull/25864）
https://issues.apache.org/jira/browse/SPARK-29286 （
https://github.com/apache/spark/pull/26021）
https://issues.apache.org/jira/browse/SPARK-29286
<https://github.com/apache/spark/pull/26636>   (
https://github.com/apache/spark/pull/26636) --- this one is in progress

[2]:
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
     The job spark-master-test-maven-arm same with community x86 job
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/
,
     It runs all java/scala tests, the test number is about 21,112.
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-python-arm/
     The job spark-master-test-python-arm runs pyspark tests with python3.6.