You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/02 05:44:08 UTC
[GitHub] [spark] LuciferYang opened a new pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
LuciferYang opened a new pull request #32648:
URL: https://github.com/apache/spark/pull/32648
### What changes were proposed in this pull request?
This PR aims to update from Scala 2.13.5 to Scala 2.13.6 for Apache Spark 3.2.
### Why are the changes needed?
Scala 2.13.6 is a maintenance release for 2.13 line and the release notes as follows:
- [https://github.com/scala/scala/releases/tag/v2.13.6](https://github.com/scala/scala/releases/tag/v2.13.6)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the GitHub Action Scala 2.13 job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847635345
close this pr and skip Scala 2.13.6 because of [https://github.com/scala/bug/issues/12403](https://github.com/scala/bug/issues/12403), thanks @maropu @dongjoon-hyun @som-snytt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638594054
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -222,13 +222,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession with SQLHelper
// Create a test case to ignore this case.
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
- case udfTestCase: UDFTest
+ case udfTestCase: SQLQueryTestSuite#UDFTest
Review comment:
@som-snytt It seems that this compilation fix is related [https://github.com/scala/scala/pull/9504](https://github.com/scala/scala/pull/9504).
So I have another question to ask you for help, `UDFTest` is an internal `trait` defined in `SQLQueryTestSuite`, must full path be used for this case in future Scala versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846882817
maven build can pass, but sbt build failed
```
[error] java.util.NoSuchElementException: None.get
[error] at scala.None$.get(Option.scala:627)
[error] at scala.None$.get(Option.scala:626)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:163)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:160)
[error] at com.typesafe.genjavadoc.BasicTransform.withClass(BasicTransform.scala:129)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:91)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:93)
[error] at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2622)
[error] at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2620)
[error] at scala.reflect.internal.Trees$PackageDef.$anonfun$transform$1(Trees.scala:342)
[error] at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2633)
[error] at scala.reflect.internal.Trees$PackageDef.transform(Trees.scala:342)
[error] at scala.reflect.internal.Trees$InternalTransformer.transform(Trees.scala:1489)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransform(Plugin.scala:102)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:115)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:183)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransformUnit(Plugin.scala:101)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit(BasicTransform.scala:20)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit$(BasicTransform.scala:19)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransformUnit(Plugin.scala:93)
```
It looks like you need to wait for `com.typesafe.genjavadoc` to release a new version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960890363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions.
To prove this I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars`
2.Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I thinkd we can ignore it from this PR and I will create a jira to continue tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740814443
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60: fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60:
fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
cc @dongjoon-hyun This error report a little strange
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
Add a rule suppress this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960432412
The full UTs passed with Scala 2.13.7 and let me verify the UTs again for double check, will give feedback later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957264004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144835/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725170
##########
File path: core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
##########
@@ -283,7 +283,7 @@ private[spark] class DiskBlockObjectWriter(
}
// For testing
- private[spark] override def flush(): Unit = {
+ override def flush(): Unit = {
Review comment:
[ERROR] [Error] /spark/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala:286: weaker access privileges in overriding
def flush(): Unit (defined in class OutputStream)
override should be public
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637734438
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
##########
@@ -107,10 +107,10 @@ class ThriftServerQueryTestSuite extends SQLQueryTestSuite with SharedThriftServ
}
testCase match {
- case _: PgSQLTest =>
+ case _: SQLQueryTestSuite#PgSQLTest =>
Review comment:
```
/spark/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala:110:15
The outer reference in this type test cannot be checked at run time.
case _: PgSQLTest =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638583811
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochTracker.scala
##########
@@ -28,7 +28,7 @@ object EpochTracker {
// update the underlying AtomicLong as it finishes epochs. Other code should only read the value.
private val currentEpoch: InheritableThreadLocal[AtomicLong] = {
new InheritableThreadLocal[AtomicLong] {
- override protected def childValue(parent: AtomicLong): AtomicLong = {
+ override def childValue(parent: AtomicLong): AtomicLong = {
Review comment:
It seems that this change is due to [https://github.com/scala/scala/pull/9542](https://github.com/scala/scala/pull/9542)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725329
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -352,7 +352,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Thread Local variable that can be used by users to pass information down the stack
protected[spark] val localProperties = new InheritableThreadLocal[Properties] {
- override protected def childValue(parent: Properties): Properties = {
+ override def childValue(parent: Properties): Properties = {
Review comment:
[ERROR] [Error] /spark/core/src/main/scala/org/apache/spark/SparkContext.scala:355: weaker access privileges in overriding
private[package lang] def childValue(x$1: java.util.Properties): java.util.Properties (defined in class ThreadLocal)
override should at least be private[lang]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638582161
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -81,6 +81,7 @@ class Iso8601DateFormatter(
try {
formatter
} catch checkLegacyFormatter(pattern, legacyFormatter.validatePatternString)
+ ()
Review comment:
thx ~ @som-snytt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846908861
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960890363
but if we run `clean test` or `clean install` directly as follows(both Scala-2.12 & Scala-2.13):
```
build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
```
or
```
build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
```
There are 3 tests failed in `resource-managers/yarn` module:
```
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
```
The error stack is
```
21/11/04 19:48:52.159 main ERROR Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times,
most recent failure: Lost task 0.3 in stage 0.0 (TID 6) (localhost executor 1): java.lang.NoClassDefFoundError: breeze/linalg/Matrix
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:216)
at org.apache.spark.serializer.KryoSerializer$.$anonfun$loadableSparkClasses$1(KryoSerializer.scala:537)
at scala.collection.immutable.List.flatMap(List.scala:293)
at scala.collection.immutable.List.flatMap(List.scala:79)
at org.apache.spark.serializer.KryoSerializer$.loadableSparkClasses$lzycompute(KryoSerializer.scala:535)
at org.apache.spark.serializer.KryoSerializer$.org$apache$spark$serializer$KryoSerializer$$loadableSparkClasses(KryoSerializer.scala:502)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:226)
at org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
at org.apache.spark.serializer.KryoSerializer$PoolWrapper.borrow(KryoSerializer.scala:109)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:346)
at org.apache.spark.serializer.KryoSerializationStream.<init>(KryoSerializer.scala:266)
at org.apache.spark.serializer.KryoSerializerInstance.serializeStream(KryoSerializer.scala:432)
at org.apache.spark.shuffle.ShufflePartitionPairsWriter.open(ShufflePartitionPairsWriter.scala:76)
at org.apache.spark.shuffle.ShufflePartitionPairsWriter.write(ShufflePartitionPairsWriter.scala:59)
at org.apache.spark.util.collection.WritablePartitionedIterator.writeNext(WritablePartitionedPairCollection.scala:83)
at org.apache.spark.util.collection.ExternalSorter.$anonfun$writePartitionedMapOutput$1(ExternalSorter.scala:772)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.util.collection.ExternalSorter.writePartitionedMapOutput(ExternalSorter.scala:775)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: breeze.linalg.Matrix
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
```
And I don't think this is related to the current pr. this problem can be reproduced in the master. I will create new a PR to investigate it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961193305
And I upload `success-unit-tests.log` and `failed-unit-tests.log` to SPARK-37209 to facilitate troubleshooting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957313007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957321231
**[Test build #144843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144843/testReport)** for PR 32648 at commit [`700e388`](https://github.com/apache/spark/commit/700e38807155e80eeac42c0cd801c071746ffcb0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846806014
This patch is still self testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847825748
**[Test build #138928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138928/testReport)** for PR 32648 at commit [`2a4d711`](https://github.com/apache/spark/commit/2a4d7110afd9f0c40dec14eef63dbd9d9f153649).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847500606
**[Test build #138902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138902/testReport)** for PR 32648 at commit [`c25bc9a`](https://github.com/apache/spark/commit/c25bc9a3c73176539d21e6ad25066bd93b920ffe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846826459
**[Test build #138862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138862/testReport)** for PR 32648 at commit [`b46c58a`](https://github.com/apache/spark/commit/b46c58a861a87ae6d033edd536f169fdea100f70).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957279058
**[Test build #144838 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144838/testReport)** for PR 32648 at commit [`08e49df`](https://github.com/apache/spark/commit/08e49df5b44d2e96980bcf430f2aa9cf1a466e16).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957348275
**[Test build #144844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144844/testReport)** for PR 32648 at commit [`56d3956`](https://github.com/apache/spark/commit/56d395668839188dc6ea4fb95a96a1b2950b52e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960980974
Hm, that's weird. So this is a pre-existing problem, appears with 2.12 too? I wonder why we haven't seen it before though.
I think the issue is that Kryo pre-registers a bunch of classes, including some from Spark ML. YARN doesn't depend on Spark ML though, so it doesn't see breeze. (But I wonder why this hasn't arisen before.) I could imagine it happens to work out somehow if you pre-build Spark entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-962228829
Merged to master for Apache Spark 3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846864643
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725520
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -81,6 +81,7 @@ class Iso8601DateFormatter(
try {
formatter
} catch checkLegacyFormatter(pattern, legacyFormatter.validatePatternString)
+ ()
Review comment:
```
[ERROR] [Error] /spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala:83: type mismatch;
found : PartialFunction[Throwable,java.time.format.DateTimeFormatter]
required: Throwable => Unit
[INFO] [Info] : PartialFunction[Throwable,java.time.format.DateTimeFormatter] <: Throwable => Unit?
[INFO] [Info] : false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] som-snytt commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
som-snytt commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638645264
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -222,13 +222,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession with SQLHelper
// Create a test case to ignore this case.
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
- case udfTestCase: UDFTest
+ case udfTestCase: SQLQueryTestSuite#UDFTest
Review comment:
I don't have a precise answer, but I would expect ongoing improvements and also some noise. You can use `-Wconf` to suppress (or escalate) the message. I will try to take another look tomorrow; I haven't followed the pattern matching changes yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638746878
##########
File path: pom.xml
##########
@@ -3387,6 +3387,8 @@
<arg>-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s</arg>
<arg>-Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s</arg>
<arg>-Wconf:msg=method without a parameter list overrides a method with a single empty one:s</arg>
+ <!-- SPARK-35496 Suppress `The outer reference in this type test cannot be checked at run time` -->
Review comment:
should be `<arg>-Wconf:cat=unchecked&msg=outer reference:s</arg>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846908861
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847825748
**[Test build #138928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138928/testReport)** for PR 32648 at commit [`2a4d711`](https://github.com/apache/spark/commit/2a4d7110afd9f0c40dec14eef63dbd9d9f153649).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847516624
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638513891
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
Thank you for filing the upstream issue. Ya, it looks like a breaking behavior change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847938906
**[Test build #138928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138928/testReport)** for PR 32648 at commit [`2a4d711`](https://github.com/apache/spark/commit/2a4d7110afd9f0c40dec14eef63dbd9d9f153649).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957221942
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957226613
**[Test build #144844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144844/testReport)** for PR 32648 at commit [`56d3956`](https://github.com/apache/spark/commit/56d395668839188dc6ea4fb95a96a1b2950b52e8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
**[Test build #144916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144916/testReport)** for PR 32648 at commit [`2481e1d`](https://github.com/apache/spark/commit/2481e1db6d96ea0833af02641905d686449682bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957354914
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144844/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957138916
**[Test build #144838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144838/testReport)** for PR 32648 at commit [`08e49df`](https://github.com/apache/spark/commit/08e49df5b44d2e96980bcf430f2aa9cf1a466e16).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740814443
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60: fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60:
fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
cc @dongjoon-hyun This error report a little strange
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
Add a rule suppress this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960890363
but if we run `clean test` or `clean install` directly as follows:
```
build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
```
or
```
build/mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
```
There are 3 tests failed in `resource-managers/yarn` module:
```
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
- external shuffle service *** FAILED ***
FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:227)
```
The error stack is
```
21/11/04 19:48:52.159 main ERROR Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times,
most recent failure: Lost task 0.3 in stage 0.0 (TID 6) (localhost executor 1): java.lang.NoClassDefFoundError: breeze/linalg/Matrix
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:216)
at org.apache.spark.serializer.KryoSerializer$.$anonfun$loadableSparkClasses$1(KryoSerializer.scala:537)
at scala.collection.immutable.List.flatMap(List.scala:293)
at scala.collection.immutable.List.flatMap(List.scala:79)
at org.apache.spark.serializer.KryoSerializer$.loadableSparkClasses$lzycompute(KryoSerializer.scala:535)
at org.apache.spark.serializer.KryoSerializer$.org$apache$spark$serializer$KryoSerializer$$loadableSparkClasses(KryoSerializer.scala:502)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:226)
at org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
at org.apache.spark.serializer.KryoSerializer$PoolWrapper.borrow(KryoSerializer.scala:109)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:346)
at org.apache.spark.serializer.KryoSerializationStream.<init>(KryoSerializer.scala:266)
at org.apache.spark.serializer.KryoSerializerInstance.serializeStream(KryoSerializer.scala:432)
at org.apache.spark.shuffle.ShufflePartitionPairsWriter.open(ShufflePartitionPairsWriter.scala:76)
at org.apache.spark.shuffle.ShufflePartitionPairsWriter.write(ShufflePartitionPairsWriter.scala:59)
at org.apache.spark.util.collection.WritablePartitionedIterator.writeNext(WritablePartitionedPairCollection.scala:83)
at org.apache.spark.util.collection.ExternalSorter.$anonfun$writePartitionedMapOutput$1(ExternalSorter.scala:772)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.util.collection.ExternalSorter.writePartitionedMapOutput(ExternalSorter.scala:775)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: breeze.linalg.Matrix
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions.
To prove this I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars`
2. Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I think we can ignore it from this PR and I created a new jira(SPARK-37209) to continue tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #32648:
URL: https://github.com/apache/spark/pull/32648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846889839
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847827116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43449/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638430999
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
I submitted an issue to consult the Scala community [https://github.com/scala/bug/issues/12403](https://github.com/scala/bug/issues/12403)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846927925
**[Test build #138862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138862/testReport)** for PR 32648 at commit [`b46c58a`](https://github.com/apache/spark/commit/b46c58a861a87ae6d033edd536f169fdea100f70).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725520
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -81,6 +81,7 @@ class Iso8601DateFormatter(
try {
formatter
} catch checkLegacyFormatter(pattern, legacyFormatter.validatePatternString)
+ ()
Review comment:
ERROR] [Error] /spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala:83: type mismatch;
found : PartialFunction[Throwable,java.time.format.DateTimeFormatter]
required: Throwable => Unit
[INFO] [Info] : PartialFunction[Throwable,java.time.format.DateTimeFormatter] <: Throwable => Unit?
[INFO] [Info] : false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-848407585
@som-snytt Thank you for your answer, use `-Wconf:cat=unchecked&msg=outer reference:s` can suppress this compilation problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638426438
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
In Scala 2.13.5
```
Welcome to Scala 2.13.5 (OpenJDK 64-Bit Server VM, Java 1.8.0_232).
Type in expressions for evaluation. Or try :help.
scala> Array.empty[Double].intersect(Array(0.0))
val res0: Array[Double] = Array()
scala> Array(0.0).intersect(Array.empty[Double])
val res1: Array[Double] = Array()
```
but in Scala 2.13.6
```
Welcome to Scala 2.13.6 (OpenJDK 64-Bit Server VM, Java 1.8.0_232).
Type in expressions for evaluation. Or try :help.
scala> Array.empty[Double].intersect(Array(0.0))
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [D
... 32 elided
scala> Array(0.0).intersect(Array.empty[Double])
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [D
... 32 elided
```
`Array()` is replaced by throw `ClassCastException`, this change is uncomfortable ... @dongjoon-hyun @maropu do you think this is a bug of Scala 2.13.6?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957221789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957313007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957143596
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957263016
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49313/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957322283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957142554
There are some new compilation errors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957252300
**[Test build #144835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144835/testReport)** for PR 32648 at commit [`c035ca9`](https://github.com/apache/spark/commit/c035ca92908762dfbd4e83c19a93f29101334dea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
**[Test build #144916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144916/testReport)** for PR 32648 at commit [`2481e1d`](https://github.com/apache/spark/commit/2481e1db6d96ea0833af02641905d686449682bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
**[Test build #144916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144916/testReport)** for PR 32648 at commit [`2481e1d`](https://github.com/apache/spark/commit/2481e1db6d96ea0833af02641905d686449682bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] som-snytt commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
som-snytt commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638564182
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -81,6 +81,7 @@ class Iso8601DateFormatter(
try {
formatter
} catch checkLegacyFormatter(pattern, legacyFormatter.validatePatternString)
+ ()
Review comment:
This aligns with Scala 3. I guess the `Unit` here comes from the method result type. I'll ask whether that is desirable; arguably it could insert the parens automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637726936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -222,13 +222,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession with SQLHelper
// Create a test case to ignore this case.
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
- case udfTestCase: UDFTest
+ case udfTestCase: SQLQueryTestSuite#UDFTest
Review comment:
```
[ERROR] [Error] /spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:225: The outer reference in this type test cannot be checked at run time.
[ERROR] [Error] /spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:357: The outer reference in this type test cannot be checked at run time.
[ERROR] [Error] /spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:363: The outer reference in this type test cannot be checked at run time.
[ERROR] [Error] /spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:371: The outer reference in this type test cannot be checked at run time.
[ERROR] [Error] /spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala:414: The outer reference in this type test cannot be checked at run time.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847542864
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638518735
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
As @LuciferYang mentioned at the Scala community, I also confirmed that only Scala 2.13.6 has this bug unlike Scala 2.13.5/3.0.0.
```
scala3-3.0.0:$ bin/scala
scala> Array.empty[Double].intersect(Array(0.0))
val res0: Array[Double] = Array()
scala-2.13.6:$ bin/scala
Welcome to Scala 2.13.6 (OpenJDK 64-Bit Server VM, Java 1.8.0_292).
Type in expressions for evaluation. Or try :help.
scala> Array.empty[Double].intersect(Array(0.0))
java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [D
... 32 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846882817
maven build can pass, but sbt build failed
```
[error] java.util.NoSuchElementException: None.get
[error] at scala.None$.get(Option.scala:627)
[error] at scala.None$.get(Option.scala:626)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:163)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:160)
[error] at com.typesafe.genjavadoc.BasicTransform.withClass(BasicTransform.scala:129)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:91)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:93)
[error] at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2622)
[error] at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2620)
[error] at scala.reflect.internal.Trees$PackageDef.$anonfun$transform$1(Trees.scala:342)
[error] at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2633)
[error] at scala.reflect.internal.Trees$PackageDef.transform(Trees.scala:342)
[error] at scala.reflect.internal.Trees$InternalTransformer.transform(Trees.scala:1489)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransform(Plugin.scala:102)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:115)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:183)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransformUnit(Plugin.scala:101)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit(BasicTransform.scala:20)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit$(BasicTransform.scala:19)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransformUnit(Plugin.scala:93)
```
It looks like we need to wait for `com.typesafe.genjavadoc` to release a new version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847479793
> BTW, could you summarize the required changes in the PR description?
It seems that those changes will affect the end-users Spark application, too.
OK， I will do this later, it seems there are some UTs in Scala 2.13 failed, I'll fix these first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957354914
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144844/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168841
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957174455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957174455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957223499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144834/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957137310
**[Test build #144835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144835/testReport)** for PR 32648 at commit [`c035ca9`](https://github.com/apache/spark/commit/c035ca92908762dfbd4e83c19a93f29101334dea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740814443
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60: fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961661216
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144916/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961823307
@srowen @dongjoon-hyun Manual test 2481e1d with Scala 2.13 passed
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.3.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.095 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 2.485 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 7.960 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 2.630 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 50.558 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.483 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 2.044 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 1.909 s]
[INFO] Spark Project Core ................................. SUCCESS [26:10 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 18.530 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:28 min]
[INFO] Spark Project Streaming ............................ SUCCESS [04:35 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [16:37 min]
[INFO] Spark Project SQL .................................. SUCCESS [ 01:10 h]
[INFO] Spark Project ML Library ........................... SUCCESS [16:58 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 0.792 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 01:33 h]
[INFO] Spark Project REPL ................................. SUCCESS [01:32 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 0.933 s]
[INFO] Spark Project YARN ................................. SUCCESS [07:40 min]
[INFO] Spark Project Mesos ................................ SUCCESS [ 47.649 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [02:02 min]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [23:54 min]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 5.807 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 14.767 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 2.372 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 33.524 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:54 min]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [37:30 min]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 58.748 s]
[INFO] Spark Project Examples ............................. SUCCESS [03:12 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 1.058 s]
[INFO] Spark Avro ......................................... SUCCESS [03:35 min]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 1.172 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:15 h
[INFO] Finished at: 2021-11-05T19:29:09+08:00
[INFO] ------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957142554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846826459
**[Test build #138862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138862/testReport)** for PR 32648 at commit [`b46c58a`](https://github.com/apache/spark/commit/b46c58a861a87ae6d033edd536f169fdea100f70).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961642204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960440605
Thank you, @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957142554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions.
To prove this I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars`
2. Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I think we can ignore it from this PR and I created a new jira(https://issues.apache.org/jira/browse/SPARK-37209) to continue tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957138916
**[Test build #144838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144838/testReport)** for PR 32648 at commit [`08e49df`](https://github.com/apache/spark/commit/08e49df5b44d2e96980bcf430f2aa9cf1a466e16).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740814443
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
```
[ERROR] [Error] /spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala:60:
fruitless type test: a value of type org.apache.spark.deploy.k8s.submit.MainAppResource cannot also be a org.apache.spark.deploy.k8s.submit.NonJVMResource
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957274760
Let me check UTs manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957174421
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957322283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961161047
Thank you for reporting, @LuciferYang . I also agree with @srowen 's guess.
If this is the same on Scala 2.13.5 or Scala 2.12.x, we can ignore it from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
**[Test build #144834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144834/testReport)** for PR 32648 at commit [`ece1339`](https://github.com/apache/spark/commit/ece1339616aa646004739706b389ede6fd6ed32e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961624056
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49387/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847542864
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang closed pull request #32648:
URL: https://github.com/apache/spark/pull/32648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847827116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43449/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638654223
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -222,13 +222,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession with SQLHelper
// Create a test case to ignore this case.
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
- case udfTestCase: UDFTest
+ case udfTestCase: SQLQueryTestSuite#UDFTest
Review comment:
thanks, I tried to suppress this behavior in the past, but it seems that this is a compilation error now, I'll try it again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638519358
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
In this case, we had better skip Scala 2.13.6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847577153
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725170
##########
File path: core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
##########
@@ -283,7 +283,7 @@ private[spark] class DiskBlockObjectWriter(
}
// For testing
- private[spark] override def flush(): Unit = {
+ override def flush(): Unit = {
Review comment:
```
[ERROR] [Error] /spark/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala:286: weaker access privileges in overriding
def flush(): Unit (defined in class OutputStream)
override should be public
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957223499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144834/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957264004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144835/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957266612
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847577153
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions.
To prove the I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars`
2.Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I thinkd we can ignore it from this PR and I will create a jira to continue tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637726014
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -107,6 +107,7 @@ class Iso8601TimestampFormatter(
try {
formatter
} catch checkLegacyFormatter(pattern, legacyFormatter.validatePatternString)
+ ()
Review comment:
```
[ERROR] [Error] /spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala:109: type mismatch;
found : PartialFunction[Throwable,java.time.format.DateTimeFormatter]
required: Throwable => Unit
[INFO] [Info] : PartialFunction[Throwable,java.time.format.DateTimeFormatter] <: Throwable => Unit?
[INFO] [Info] : false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846882817
maven build can pass, but sbt build failed
```
[error] java.util.NoSuchElementException: None.get
[error] at scala.None$.get(Option.scala:627)
[error] at scala.None$.get(Option.scala:626)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:163)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:160)
[error] at com.typesafe.genjavadoc.BasicTransform.withClass(BasicTransform.scala:129)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:91)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:93)
[error] at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2622)
[error] at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2620)
[error] at scala.reflect.internal.Trees$PackageDef.$anonfun$transform$1(Trees.scala:342)
[error] at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2633)
[error] at scala.reflect.internal.Trees$PackageDef.transform(Trees.scala:342)
[error] at scala.reflect.internal.Trees$InternalTransformer.transform(Trees.scala:1489)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransform(Plugin.scala:102)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:115)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:183)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransformUnit(Plugin.scala:101)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit(BasicTransform.scala:20)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit$(BasicTransform.scala:19)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransformUnit(Plugin.scala:93)
```
**It looks like we need to wait for `com.typesafe.genjavadoc` to release a new version**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847968949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138928/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957303200
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49314/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740815797
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
cc @dongjoon-hyun This error report a little strange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957221047
**[Test build #144843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144843/testReport)** for PR 32648 at commit [`700e388`](https://github.com/apache/spark/commit/700e38807155e80eeac42c0cd801c071746ffcb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846943561
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961647678
**[Test build #144916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144916/testReport)** for PR 32648 at commit [`2481e1d`](https://github.com/apache/spark/commit/2481e1db6d96ea0833af02641905d686449682bb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait ExpressionBuilder `
* `trait PadExpressionBuilderBase extends ExpressionBuilder `
* `case class StringLPad(str: Expression, len: Expression, pad: Expression)`
* `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class DropIndex(`
* `public class ColumnIOUtil `
* `case class ParquetColumn(`
* `case class DropIndexExec(`
* `case class PushedDownOperators(`
* `case class TableSampleInfo(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961605563
**[Test build #144916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144916/testReport)** for PR 32648 at commit [`2481e1d`](https://github.com/apache/spark/commit/2481e1db6d96ea0833af02641905d686449682bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961593647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637729610
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -509,7 +509,7 @@ case class NoCloseColumnVector(wrapped: ColumnVector) extends ColumnVector(wrapp
override def getBinary(rowId: Int): Array[Byte] = wrapped.getBinary(rowId)
- override protected def getChild(ordinal: Int): ColumnVector = wrapped.getChild(ordinal)
+ override def getChild(ordinal: Int): ColumnVector = wrapped.getChild(ordinal)
Review comment:
```
/spark/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala:512:26
weaker access privileges in overriding
def getChild(x$1: Int): org.apache.spark.sql.vectorized.ColumnVector (defined in class ColumnVector)
override should be public
override protected def getChild(ordinal: Int): ColumnVector = wrapped.getChild(ordinal)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637725329
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -352,7 +352,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Thread Local variable that can be used by users to pass information down the stack
protected[spark] val localProperties = new InheritableThreadLocal[Properties] {
- override protected def childValue(parent: Properties): Properties = {
+ override def childValue(parent: Properties): Properties = {
Review comment:
```
[ERROR] [Error] /spark/core/src/main/scala/org/apache/spark/SparkContext.scala:355: weaker access privileges in overriding
private[package lang] def childValue(x$1: java.util.Properties): java.util.Properties (defined in class ThreadLocal)
override should at least be private[lang]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637734792
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
##########
@@ -107,10 +107,10 @@ class ThriftServerQueryTestSuite extends SQLQueryTestSuite with SharedThriftServ
}
testCase match {
- case _: PgSQLTest =>
+ case _: SQLQueryTestSuite#PgSQLTest =>
statement.execute(s"SET ${SQLConf.ANSI_ENABLED.key} = true")
statement.execute(s"SET ${SQLConf.LEGACY_INTERVAL_ENABLED.key} = true")
- case _: AnsiTest =>
+ case _: SQLQueryTestSuite#AnsiTest =>
Review comment:
```
/spark/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala:113:15
The outer reference in this type test cannot be checked at run time.
case _: AnsiTest =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846882817
maven build can pass, but sbt build failed
```
[error] java.util.NoSuchElementException: None.get
[error] at scala.None$.get(Option.scala:627)
[error] at scala.None$.get(Option.scala:626)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:163)
[error] at com.typesafe.genjavadoc.BasicTransform.deprecationInfo(BasicTransform.scala:160)
[error] at com.typesafe.genjavadoc.BasicTransform.withClass(BasicTransform.scala:129)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:91)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:93)
[error] at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2622)
[error] at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2620)
[error] at scala.reflect.internal.Trees$PackageDef.$anonfun$transform$1(Trees.scala:342)
[error] at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2633)
[error] at scala.reflect.internal.Trees$PackageDef.transform(Trees.scala:342)
[error] at scala.reflect.internal.Trees$InternalTransformer.transform(Trees.scala:1489)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransform(Plugin.scala:102)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform(BasicTransform.scala:115)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransform$(BasicTransform.scala:49)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransform(Plugin.scala:93)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.transform(Plugin.scala:103)
[error] at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:183)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.superTransformUnit(Plugin.scala:101)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit(BasicTransform.scala:20)
[error] at com.typesafe.genjavadoc.BasicTransform.newTransformUnit$(BasicTransform.scala:19)
[error] at com.typesafe.genjavadoc.GenJavadocPlugin$MyComponent$GenJavadocTransformer.newTransformUnit(Plugin.scala:93)
```
**It looks like we need to wait for `genjavadoc` to release a new version**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638743793
##########
File path: pom.xml
##########
@@ -3387,6 +3387,8 @@
<arg>-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s</arg>
<arg>-Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s</arg>
<arg>-Wconf:msg=method without a parameter list overrides a method with a single empty one:s</arg>
+ <!-- SPARK-35496 Suppress `The outer reference in this type test cannot be checked at run time` -->
Review comment:
Just to try not to modify SQLQueryTestSuite and ThriftServerQueryTestSuite, it may be useful to upgrade Scala 2.13.7 later
##########
File path: pom.xml
##########
@@ -3387,6 +3387,8 @@
<arg>-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s</arg>
<arg>-Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s</arg>
<arg>-Wconf:msg=method without a parameter list overrides a method with a single empty one:s</arg>
+ <!-- SPARK-35496 Suppress `The outer reference in this type test cannot be checked at run time` -->
Review comment:
Just to try not to change SQLQueryTestSuite and ThriftServerQueryTestSuite, it may be useful to upgrade Scala 2.13.7 later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r638527867
##########
File path: mllib/src/main/scala/org/apache/spark/mllib/evaluation/MultilabelMetrics.scala
##########
@@ -178,7 +178,11 @@ private[evaluation] class MultilabelSummarizer extends Serializable {
* @return This MultilabelSummarizer object.
*/
def add(predictions: Array[Double], labels: Array[Double]): this.type = {
- val intersection = predictions.intersect(labels)
+ val intersection = if (predictions.isEmpty || labels.isEmpty) {
Review comment:
@dongjoon-hyun agree with you. we'd better skip this version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r637726282
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochTracker.scala
##########
@@ -28,7 +28,7 @@ object EpochTracker {
// update the underlying AtomicLong as it finishes epochs. Other code should only read the value.
private val currentEpoch: InheritableThreadLocal[AtomicLong] = {
new InheritableThreadLocal[AtomicLong] {
- override protected def childValue(parent: AtomicLong): AtomicLong = {
+ override def childValue(parent: AtomicLong): AtomicLong = {
Review comment:
```
[ERROR] [Error] /spark/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochTracker.scala:31: weaker access privileges in overriding
private[package lang] def childValue(x$1: java.util.concurrent.atomic.AtomicLong): java.util.concurrent.atomic.AtomicLong (defined in class ThreadLocal)
override should at least be private[lang]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957167970
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957308252
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49313/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957139201
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
**[Test build #144834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144834/testReport)** for PR 32648 at commit [`ece1339`](https://github.com/apache/spark/commit/ece1339616aa646004739706b389ede6fd6ed32e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957221942
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32648:
URL: https://github.com/apache/spark/pull/32648#discussion_r740822929
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala
##########
@@ -57,7 +57,8 @@ private[spark] class BasicDriverFeatureStep(conf: KubernetesDriverConf)
// The memory overhead factor to use. If the user has not set it, then use a different
// value for non-JVM apps. This value is propagated to executors.
private val overheadFactor =
- if (conf.mainAppResource.isInstanceOf[NonJVMResource]) {
+ if (conf.mainAppResource.isInstanceOf[PythonMainAppResource] ||
Review comment:
Add a rule suppress this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions.
To prove this I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars`
2. Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I thinkd we can ignore it from this PR and I will create a jira to continue tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961642204
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49387/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-962229215
Thank you, @LuciferYang , @srowen , @som-snytt , @sarutak .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-962391536
thanks all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847575723
**[Test build #138902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138902/testReport)** for PR 32648 at commit [`c25bc9a`](https://github.com/apache/spark/commit/c25bc9a3c73176539d21e6ad25066bd93b920ffe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847528088
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847968949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138928/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] som-snytt commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
som-snytt commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847756599
Sample for wconf, where `i` says "info", `s` would be silence. It warns about deprecation but outer check is just info.
```
➜ snips scalac -d /tmp -Xlint "-Wconf:cat=unchecked&msg=outer reference:i" -Werror badcheck.scala
badcheck.scala:11: warning: method g in class C is deprecated (since 0.1): bad g
def x = g + 1
^
badcheck.scala:7: The outer reference in this type test cannot be checked at run time.
case _: U => true
^
error: No warnings can be incurred under -Werror.
1 warning
1 error
➜ snips cat badcheck.scala
class C {
trait T
trait U
def f(t: T) = t match {
case _: U => true
case _ => false
}
@deprecated("bad g", "0.1") def g = 42
def x = g + 1
}
```
I have to re-read `-Wconf:help` if I use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-847500606
**[Test build #138902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138902/testReport)** for PR 32648 at commit [`c25bc9a`](https://github.com/apache/spark/commit/c25bc9a3c73176539d21e6ad25066bd93b920ffe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961593647
2481e1d merge master into this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961593647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961642204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-960878693
@srowen @dongjoon-hyun Run the UTs twice using the following command and passed.
```
build/mvn clean install -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
```
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.3.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.265 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 7.661 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 13.704 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.779 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 53.708 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 10.034 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 9.294 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.462 s]
[INFO] Spark Project Core ................................. SUCCESS [25:13 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 23.937 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:42 min]
[INFO] Spark Project Streaming ............................ SUCCESS [05:25 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [18:05 min]
[INFO] Spark Project SQL .................................. SUCCESS [ 01:12 h]
[INFO] Spark Project ML Library ........................... SUCCESS [18:50 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 7.607 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 01:29 h]
[INFO] Spark Project REPL ................................. SUCCESS [01:25 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 11.999 s]
[INFO] Spark Project YARN ................................. SUCCESS [08:15 min]
[INFO] Spark Project Mesos ................................ SUCCESS [01:24 min]
[INFO] Spark Project Kubernetes ........................... SUCCESS [03:10 min]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [22:29 min]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 8.798 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 19.037 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 2.683 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 34.654 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [01:57 min]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [37:22 min]
[INFO] Spark Kinesis Integration .......................... SUCCESS [01:03 min]
[INFO] Spark Project Examples ............................. SUCCESS [03:24 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 14.442 s]
[INFO] Spark Avro ......................................... SUCCESS [03:34 min]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 17.394 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:19 h
[INFO] Finished at: 2021-11-04T16:53:26+08:00
[INFO] ------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961642188
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49387/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957221047
**[Test build #144843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144843/testReport)** for PR 32648 at commit [`700e388`](https://github.com/apache/spark/commit/700e38807155e80eeac42c0cd801c071746ffcb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957137310
**[Test build #144835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144835/testReport)** for PR 32648 at commit [`c035ca9`](https://github.com/apache/spark/commit/c035ca92908762dfbd4e83c19a93f29101334dea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957119920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957168866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32648: [SPARK-35496][BUILD] Upgrade Scala to 2.13.7
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-957226613
**[Test build #144844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144844/testReport)** for PR 32648 at commit [`56d3956`](https://github.com/apache/spark/commit/56d395668839188dc6ea4fb95a96a1b2950b52e8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32648: [WIP][SPARK-35496][BUILD] Upgrade Scala to 2.13.6
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-846943561
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org