You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2020/01/17 00:11:02 UTC
[spark] branch branch-2.4 updated: [SPARK-30534][INFRA] Use mvn in
`dev/scalastyle`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ac54156 [SPARK-30534][INFRA] Use mvn in `dev/scalastyle`
ac54156 is described below
commit ac541568d69388a3fd6467e8792900840573a590
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Thu Jan 16 16:00:58 2020 -0800
[SPARK-30534][INFRA] Use mvn in `dev/scalastyle`
This PR aims to use `mvn` instead of `sbt` in `dev/scalastyle` to recover GitHub Action.
As of now, Apache Spark sbt build is broken by the Maven Central repository policy.
https://stackoverflow.com/questions/59764749/requests-to-http-repo1-maven-org-maven2-return-a-501-https-required-status-an
> Effective January 15, 2020, The Central Maven Repository no longer supports insecure
> communication over plain HTTP and requires that all requests to the repository are
> encrypted over HTTPS.
We can reproduce this locally by the following.
```
$ rm -rf ~/.m2/repository/org/apache/apache/18/
$ build/sbt clean
```
And, in GitHub Action, `lint-scala` is the only one which is using `sbt`.
No.
First of all, GitHub Action should be recovered.
Also, manually, do the following.
**Without Scalastyle violation**
```
$ dev/scalastyle
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Using `mvn` from path: /usr/local/bin/mvn
Scalastyle checks passed.
```
**With Scalastyle violation**
```
$ dev/scalastyle
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Using `mvn` from path: /usr/local/bin/mvn
Scalastyle checks failed at following occurrences:
error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/main/scala/org/apache/spark/SparkConf.scala message=There should be no empty line separating imports in the same group. line=22 column=0
error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala message=There should be no empty line separating imports in the same group. line=22 column=0
```
Closes #27242 from dongjoon-hyun/SPARK-30534.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 384899944b25cb0abf5e71f9cc2610fecad4e8f5)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
dev/scalastyle | 19 ++++---------------
.../spark/sql/jdbc/OracleIntegrationSuite.scala | 2 +-
pom.xml | 2 +-
3 files changed, 6 insertions(+), 17 deletions(-)
diff --git a/dev/scalastyle b/dev/scalastyle
index b8053df..b54027f 100755
--- a/dev/scalastyle
+++ b/dev/scalastyle
@@ -17,21 +17,10 @@
# limitations under the License.
#
-# NOTE: echo "q" is needed because SBT prompts the user for input on encountering a build file
-# with failure (either resolution or compilation); the "q" makes SBT quit.
-ERRORS=$(echo -e "q\n" \
- | build/sbt \
- -Pkinesis-asl \
- -Pmesos \
- -Pkafka-0-8 \
- -Pkubernetes \
- -Pyarn \
- -Pflume \
- -Phive \
- -Phive-thriftserver \
- scalastyle test:scalastyle \
- | awk '{if($1~/error/)print}' \
-)
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
+
+SPARK_PROFILES=${1:-"-Pmesos -Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pdocker-integration-tests -Pkubernetes-integration-tests"}
+ERRORS=$($SCRIPT_DIR/../build/mvn $SPARK_PROFILES scalastyle:check | grep "^error file")
if test ! -z "$ERRORS"; then
echo -e "Scalastyle checks failed at following occurrences:\n$ERRORS"
diff --git a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
index 540de0f..85d4bbf 100644
--- a/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
+++ b/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
@@ -154,7 +154,7 @@ class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSQLCo
// A value with fractions from DECIMAL(3, 2) is correct:
assert(row.getDecimal(1).compareTo(BigDecimal.valueOf(1.23)) == 0)
// A value > Int.MaxValue from DECIMAL(10) is correct:
- assert(row.getDecimal(2).compareTo(BigDecimal.valueOf(9999999999l)) == 0)
+ assert(row.getDecimal(2).compareTo(BigDecimal.valueOf(9999999999L)) == 0)
}
diff --git a/pom.xml b/pom.xml
index 07f254a..1713df7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2542,7 +2542,7 @@
<configuration>
<verbose>false</verbose>
<failOnViolation>true</failOnViolation>
- <includeTestSourceDirectory>false</includeTestSourceDirectory>
+ <includeTestSourceDirectory>true</includeTestSourceDirectory>
<failOnWarning>false</failOnWarning>
<sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
<testSourceDirectory>${basedir}/src/test/scala</testSourceDirectory>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org