You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/24 02:14:31 UTC

[GitHub] [spark] LuciferYang opened a new pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

LuciferYang opened a new pull request #29857:
URL: https://github.com/apache/spark/pull/29857


   ### What changes were proposed in this pull request?
   The purpose of this pr is to partial resolve SPARK-32972, total of 50 Scala failed test cases and 3 Java failed test cases were fixed, the main change of this pr as follow:
   
   - Specified `Seq` to `scala.collection.Seq` in case match `Seq` scene and `x.asInstanceOf[Seq[T]]` scene
   
   - Use `Row.getSeq[T]` instead of `Row.getAs[Seq]` scene
   
   - Manual call `toMap` method to convert `MapView` to `Map` in Scala 2.13
   
   After this pr, only "training with sample weights"  in `RandomForestRegressorSuite` not fixed in Scala 2.13.
   
   ### Why are the changes needed?
   We need to support a Scala 2.13 build.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   - Scala 2.12: Pass the Jenkins or GitHub Action
   
   - Scala 2.13: Pass  GitHub 2.13 Build Action
   
   Do the follow:
   
   ```
   dev/change-scala-version.sh 2.13
   mvn clean install -DskipTests  -pl mllib -Pscala-2.13 -am
   mvn test -pl mllib -Pscala-2.13 -fn
   ```
   
   **Before**
   ```
   [ERROR] Errors: 
   [ERROR]   JavaVectorIndexerSuite.vectorIndexerAPI:51 » ClassCast scala.collection.conver...
   [ERROR]   JavaWord2VecSuite.testJavaWord2Vec:51 » Spark Job aborted due to stage failure...
   [ERROR]   JavaPrefixSpanSuite.runPrefixSpanSaveLoad:79 » Spark Job aborted due to stage ...
   
   Tests: succeeded 1567, failed 51, canceled 0, ignored 7, pending 0
   *** 51 TESTS FAILED ***
   
   ```
   
   **After**
   
   ```
   [INFO] Tests run: 122, Failures: 0, Errors: 0, Skipped: 0
   
   Tests: succeeded 1617, failed 1, canceled 0, ignored 7, pending 0
   *** 1 TESTS FAILED ***
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698124056


   **[Test build #129065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129065/testReport)** for PR 29857 at commit [`4f5eac5`](https://github.com/apache/spark/commit/4f5eac5a5906577837a348c9dc4b30c28bf80451).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699649320


   Merged to master


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699741364


   @dongjoon-hyun Should we consider adding some new GitHubActions to check test in Scala 2.13?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087474






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112073


   https://github.com/apache/spark/blob/0bc0e91e4015eb98bd2f4bf17da2ec7135b520a9/mllib/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala#L36-L54
   
   And I found if we change `nPoints` from 1000 to 1150, the test will pass


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698679002


   > f you like you can try making the ordering of the Map in this code deterministic to see if that does it:
   
   > val topNodesForGroup: Map[Int, LearningNode] = 
   >      nodesForGroup.keys.map(treeIdx => treeIdx -> topNodes(treeIdx)).toMap 
   > But, I don't even know if the result the test is complaining about is 'wrong'.
   > 
   
   OK ~ I will try this first and feedback later.
   
   But change 0.78 to 0.75 seems simpler, haha :)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070226






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698124056


   **[Test build #129065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129065/testReport)** for PR 29857 at commit [`4f5eac5`](https://github.com/apache/spark/commit/4f5eac5a5906577837a348c9dc4b30c28bf80451).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698151430






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698100080


   cc @dongjoon-hyun https://github.com/apache/spark/pull/29861 fix GitHub 2.13 build Action, related to k8s module, I will rebase this pr after it merged.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698479253


   For the random forest test, it's probably reasonably to simply change the tol in the last test to 0.75 like the others. I don't know why it should be higher.
   
   If you like you can try making the ordering of the Map in this code deterministic to see if that does it:
   ```
   val topNodesForGroup: Map[Int, LearningNode] = 
        nodesForGroup.keys.map(treeIdx => treeIdx -> topNodes(treeIdx)).toMap 
   ```
   
   But, I don't even know if the result the test is complaining about is 'wrong'.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070610


   Merged build finished. Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070617


   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33675/
   Test PASSed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699740416


   thx @srowen ~ 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112690






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699571003


   @srowen Is there any other problem in this pr that needs to be fixed?thx ~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112690


   @LuciferYang . I closed your  #29861 because `master` branch is already fixed two hours ago.
   - https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala#L117


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698124422






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698937917






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698111573


   cc @srowen The remaining failed case is 
   
   ```
   RandomForestRegressorSuite:
   - training with sample weights *** FAILED ***
     0.756 was not greater than or equal to 0.78 (MLTestingUtils.scala:285)
   ```
   
   https://github.com/apache/spark/blob/0bc0e91e4015eb98bd2f4bf17da2ec7135b520a9/mllib/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala#L171-L200
   
   Input `(50, 10, 0.95, 0.78)` with 
   
   ```
   MLTestingUtils.testOversamplingVsWeighting[RandomForestRegressionModel,
           RandomForestRegressor](df.as[LabeledPoint], estimator,
           MLTestingUtils.modelPredictionEquals(df, _ ~= _ relTol 0.2, tol), seed)
   ```
   failed.
   
   I found that the following `RandomForest.runBagged` behave differently for the same input in Scala 2.12 and Scala 2.13, maybe related to the follow code block:
   
   https://github.com/apache/spark/blob/0bc0e91e4015eb98bd2f4bf17da2ec7135b520a9/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L191-L215
   
   but I am not familiar with this algorithm and I not find root cause, I think we need an expert to guide how to fix it
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698906903


   **[Test build #129109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129109/testReport)** for PR 29857 at commit [`f2a26c5`](https://github.com/apache/spark/commit/f2a26c571b37b6f8c3ad169c27e73a38a67160f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen closed pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
srowen closed pull request #29857:
URL: https://github.com/apache/spark/pull/29857


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698937917






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699576181


   From the first round of code checking, may be this is the last module to fix Scala version compatibility


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698928101






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-699571003


   @srowen Is there any other problem in this pr that needs to be fixed?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698905421


   Address f2a26c5 fix RandomForestRegressorSuite.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698892347


   I think the latter option, changing the tol, is fine. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087474


   **[Test build #129054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit [`b7f2f47`](https://github.com/apache/spark/commit/b7f2f47c6c22af15478b89201fb6c685f47d66bd).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698819722


   Synchronize the test result:
   
   https://github.com/apache/spark/blob/0bc0e91e4015eb98bd2f4bf17da2ec7135b520a9/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L194-L202
   
   - **Test A：** Change to use `LinkedHashMap` to store topNodesForGroup and use `nodesForGroup.keys.toSeq.sorted` to init it was no essential impact.
   
   - **Test B：** Change to use `LinkedHashMap` to store `topNodesForGroup` and `nodesForGroup` seems that the goal of unified behavior in Scala 2.12 and 2.13 can be achieved,  but need change the tol in the last test to 0.77(the result is 0.771)
   
   - **Test C：** Only change the tol in the last test to 0.75 can workaround
   
   Which do you recommend better @srowen , B or C?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698899448


   ok ~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Pass all UTs of `mllib` module in Scala 2.13

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698906903


   **[Test build #129109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129109/testReport)** for PR 29857 at commit [`f2a26c5`](https://github.com/apache/spark/commit/f2a26c571b37b6f8c3ad169c27e73a38a67160f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698123928


   @dongjoon-hyun Address 4f5eac5 rebase master


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112073






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070610






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112073


   https://github.com/apache/spark/blob/0bc0e91e4015eb98bd2f4bf17da2ec7135b520a9/mllib/src/test/scala/org/apache/spark/ml/regression/RandomForestRegressorSuite.scala#L45-L54
   
   And I found if we change `nPoints` from 1000 to 1150, the test will pass


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698151430






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698151061


   **[Test build #129065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129065/testReport)** for PR 29857 at commit [`4f5eac5`](https://github.com/apache/spark/commit/4f5eac5a5906577837a348c9dc4b30c28bf80451).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698124422






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698479253






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698112837


   You need to rebase this branch .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070226


   **[Test build #129054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit [`b7f2f47`](https://github.com/apache/spark/commit/b7f2f47c6c22af15478b89201fb6c685f47d66bd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698070226


   **[Test build #129054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit [`b7f2f47`](https://github.com/apache/spark/commit/b7f2f47c6c22af15478b89201fb6c685f47d66bd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698100080






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698679002


   > f you like you can try making the ordering of the Map in this code deterministic to see if that does it:
   
   val topNodesForGroup: Map[Int, LearningNode] = 
        nodesForGroup.keys.map(treeIdx => treeIdx -> topNodes(treeIdx)).toMap 
   But, I don't even know if the result the test is complaining about is 'wrong'.
   
   OK ~ I will try this first and feedback later.
   
   But change 0.78 to 0.75 seems simpler, haha :)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29857:
URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org