You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/14 20:35:54 UTC

[GitHub] [spark] srowen opened a new pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

srowen opened a new pull request #29111:
URL: https://github.com/apache/spark/pull/29111


   ### What changes were proposed in this pull request?
   
   Same as https://github.com/apache/spark/pull/29078 and https://github.com/apache/spark/pull/28971 . This makes the rest of the default modules (i.e. those you get without specifying `-Pyarn` etc) compile under Scala 2.13. It does not close the JIRA, as a result. this also of course does not demonstrate that tests pass yet in 2.13.
   
   ### Why are the changes needed?
   
   Eventually, we need to support a Scala 2.13 build, perhaps in Spark 3.1.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   Existing tests. (2.13 was not tested; this is about getting it to compile without breaking 2.12)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658995370


   I can confirm that modules after REPL also compile (I commented out REPL locally). That's a good step, and I can look at getting the secondary modules compiling next.
   
   For anyone watching, I am entirely open to questions about the approach here. The changes are actually quite superficial but broad. I do want to keep an eye on issues like perf regressions - I do not expect them in 2.12 but may be an issue in 2.13 builds.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658403223






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658403223






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658408216


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125859/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658524127


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125864/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658887689






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658546568


   I think I understand the last test failures, will fix too.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658408206






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658489890






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29111:
URL: https://github.com/apache/spark/pull/29111#discussion_r454792607



##########
File path: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
##########
@@ -76,7 +76,7 @@ abstract class Estimator[M <: Model[M]] extends PipelineStage {
    * @return fitted models, matching the input parameter maps
    */
   @Since("2.0.0")
-  def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[M] = {
+  def fit(dataset: Dataset[_], paramMaps: Seq[ParamMap]): Seq[M] = {

Review comment:
       Yeah, this fixes the weird compile error (Arrays + generic types are stricter in Scala 2.13) though I don't directly see what it has to do with type M. Still, this is an API change I think MiMa will fail and I think I need another workaround for _that_. This is an obscure method that isn't even called by tests, AFAICT, so not sure it even has coverage. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29111:
URL: https://github.com/apache/spark/pull/29111#discussion_r454628725



##########
File path: examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java
##########
@@ -23,7 +23,7 @@
 import java.util.Arrays;
 import java.util.List;
 
-import scala.collection.mutable.WrappedArray;
+import scala.collection.mutable.Seq;

Review comment:
       WrappedArray is gone in 2.13; this should be an equivalent superclass

##########
File path: examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala
##########
@@ -82,7 +82,7 @@ object SparkKMeans {
     while(tempDist > convergeDist) {
       val closest = data.map (p => (closestPoint(p, kPoints), (p, 1)))
 
-      val pointStats = closest.reduceByKey{case ((p1, c1), (p2, c2)) => (p1 + p2, c1 + c2)}
+      val pointStats = closest.reduceByKey(mergeResults)

Review comment:
       Not quite sure why, but a few calls to `reduceByKey` didn't like the existing syntax in 2.13. I had to break out a typed method. `missing parameter type for expanded function`

##########
File path: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
##########
@@ -26,7 +27,7 @@ import org.apache.spark.sql.Dataset
 /**
  * Abstract class for estimators that fit models to data.
  */
-abstract class Estimator[M <: Model[M]] extends PipelineStage {
+abstract class Estimator[M <: Model[M] : ClassTag] extends PipelineStage {

Review comment:
       I don't quite get why 2.13 thinks this needs a ClassTag (and thus some subclasses), but I'm just going with it. Will see if MiMa is OK with it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998912


   For a record, this is merged because all tests passed in GitHub Action and I verified the Scala 2.13 compilation.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658402613


   **[Test build #125859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125859/testReport)** for PR 29111 at commit [`8b9f4a8`](https://github.com/apache/spark/commit/8b9f4a8c77141c8d8e47c909fda644d568f9452f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658408206


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29111:
URL: https://github.com/apache/spark/pull/29111#discussion_r454639188



##########
File path: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
##########
@@ -26,7 +27,7 @@ import org.apache.spark.sql.Dataset
 /**
  * Abstract class for estimators that fit models to data.
  */
-abstract class Estimator[M <: Model[M]] extends PipelineStage {
+abstract class Estimator[M <: Model[M] : ClassTag] extends PipelineStage {

Review comment:
       Unfortunately, MiMi seems to complain on this and a few others like this.
   ```
   [error]  * method this()Unit in class org.apache.spark.ml.Estimator does not have a correspondent in current version
   4232
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.Estimator.this")
   4233
   [error]  * method this()Unit in class org.apache.spark.ml.Predictor does not have a correspondent in current version
   4234
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.Predictor.this")
   4235
   [error]  * method this()Unit in class org.apache.spark.ml.classification.ProbabilisticClassifier does not have a correspondent in current version
   4236
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.classification.ProbabilisticClassifier.this")
   4237
   [error]  * method this()Unit in class org.apache.spark.ml.classification.Classifier does not have a correspondent in current version
   4238
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.classification.Classifier.this")
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658402613


   **[Test build #125859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125859/testReport)** for PR 29111 at commit [`8b9f4a8`](https://github.com/apache/spark/commit/8b9f4a8c77141c8d8e47c909fda644d568f9452f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658489540


   **[Test build #125864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125864/testReport)** for PR 29111 at commit [`bc74297`](https://github.com/apache/spark/commit/bc74297f72cf51c773b6abfe6dcd19c691f3dfac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29111:
URL: https://github.com/apache/spark/pull/29111#discussion_r454784921



##########
File path: mllib/src/main/scala/org/apache/spark/ml/Estimator.scala
##########
@@ -76,7 +76,7 @@ abstract class Estimator[M <: Model[M]] extends PipelineStage {
    * @return fitted models, matching the input parameter maps
    */
   @Since("2.0.0")
-  def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[M] = {
+  def fit(dataset: Dataset[_], paramMaps: Seq[ParamMap]): Seq[M] = {

Review comment:
       cc @mengxr and @gatorsmile




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998912


   For a record, this was merged because all tests passed in GitHub Action and I verified the Scala 2.13 compilation (https://github.com/apache/spark/pull/29111#pullrequestreview-449303638).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658408157


   **[Test build #125859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125859/testReport)** for PR 29111 at commit [`8b9f4a8`](https://github.com/apache/spark/commit/8b9f4a8c77141c8d8e47c909fda644d568f9452f).
    * This patch **fails MiMa tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `abstract class Estimator[M <: Model[M] : ClassTag] extends PipelineStage `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998506


   Also, here.
   ![Screen Shot 2020-07-15 at 1 41 21 PM](https://user-images.githubusercontent.com/9700541/87593815-e4910e00-c6a0-11ea-9e09-1c8b68fc8ed2.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29111:
URL: https://github.com/apache/spark/pull/29111#discussion_r454784282



##########
File path: examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala
##########
@@ -102,5 +102,10 @@ object SparkKMeans {
     kPoints.foreach(println)
     spark.stop()
   }
+
+  private def mergeResults(a: (Vector[Double], Int),
+                           b: (Vector[Double], Int)): (Vector[Double], Int) = {

Review comment:
       nit. Indentation?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998912


   For a record, this is merged because all tests passed in GitHub Action.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-659007765


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658524120






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658489540


   **[Test build #125864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125864/testReport)** for PR 29111 at commit [`bc74297`](https://github.com/apache/spark/commit/bc74297f72cf51c773b6abfe6dcd19c691f3dfac).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998288


   Hi, @srowen . You last commit passed the GitHub Action. Please see here.
   - https://github.com/apache/spark/pull/29111/commits/6390b6c46f5bf35e0c92b140bfbe12f98c35cd8f


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658489890






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658523924


   **[Test build #125864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125864/testReport)** for PR 29111 at commit [`bc74297`](https://github.com/apache/spark/commit/bc74297f72cf51c773b6abfe6dcd19c691f3dfac).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `abstract class Estimator[M <: Model[M]] extends PipelineStage `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998288


   Hi, @srowen . You last commit passed the GitHub Action. Please see here.
   - https://github.com/apache/spark/pull/29111/commits/6390b6c46f5bf35e0c92b140bfbe12f98c35cd8f
   
   ![Screen Shot 2020-07-15 at 1 44 07 PM](https://user-images.githubusercontent.com/9700541/87594100-48b3d200-c6a1-11ea-9488-1c07cb4758d8.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998912


   For a record, this is merged because all tests passed in GitHub Action and I verified the Scala 2.13 compilation (https://github.com/apache/spark/pull/29111#pullrequestreview-449303638).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658524120


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-659007765






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-659000238


   For now, GitHub Action finished in two hours if there is no congestion.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658887689






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658998506


   Also, here. The green checkbox at the commit id.
   ![Screen Shot 2020-07-15 at 1 41 21 PM](https://user-images.githubusercontent.com/9700541/87593815-e4910e00-c6a0-11ea-9e09-1c8b68fc8ed2.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658409026


   Yep, as I feared:
   
   ```
   [error]  * method this()Unit in class org.apache.spark.ml.Estimator does not have a correspondent in current version
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.Estimator.this")
   [error]  * method this()Unit in class org.apache.spark.ml.Predictor does not have a correspondent in current version
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.Predictor.this")
   [error]  * method this()Unit in class org.apache.spark.ml.classification.ProbabilisticClassifier does not have a correspondent in current version
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.classification.ProbabilisticClassifier.this")
   [error]  * method this()Unit in class org.apache.spark.ml.classification.Classifier does not have a correspondent in current version
   [error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.classification.Classifier.this")
   ```
   
   Adding a ClassTag means it implicitly has a new parameter in the bytecode and that breaks MiMa. Hm. I'll have to think about this more


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29111:
URL: https://github.com/apache/spark/pull/29111#issuecomment-658995776


   Oh erm, we didn't get a passing test after my last commit - I did double-check it passes 2.12 tests after the last change, but, should we be watching out for that or is this all down to github actions now?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #29111: [SPARK-29292][SQL][ML] Update rest of default modules (Hive, ML, etc) for Scala 2.13 compilation

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29111:
URL: https://github.com/apache/spark/pull/29111


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org