You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/22 11:45:40 UTC

[GitHub] [spark] zhengruifeng opened a new pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

zhengruifeng opened a new pull request #30889:
URL: https://github.com/apache/spark/pull/30889


   ### What changes were proposed in this pull request?
   In https://github.com/apache/spark/pull/21632/files#diff-0fdae8a6782091746ed20ea43f77b639f9c6a5f072dd2f600fcf9a7b37db4f47, a new field `rawCount` was added into `NodeData`, which cause that a tree model trained in 2.4 can not be loaded in 3.0/3.1/master
   
   ### Why are the changes needed?
   to support load old tree model in 3.0/3.1/master
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   added testsuites


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749872161


   @srowen Most of those files are the model files, I will try to reduce the number of files.
   
   > this should go into 3.0.x too?
   
   Yes,  in 3.0.1, I can not load any tree model created in 2.4.7


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bmarcott edited a comment on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
bmarcott edited a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-753745535


   @zhengruifeng @srowen Thanks for putting in the fix!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749552050


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37815/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749926528


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r596022053



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       It was reported vs 3.0.1, see JIRA. Looks like 3.0.2




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749503775


   **[Test build #133217 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133217/testReport)** for PR 30889 at commit [`8dae8fc`](https://github.com/apache/spark/commit/8dae8fce7894fc29187fbd95b9c0b9f3376a321e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen closed pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
srowen closed pull request #30889:
URL: https://github.com/apache/spark/pull/30889


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749517938






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-753718038


   Thanks @srowen @bmarcott  for reviewing!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749568584


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37815/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749502204


   **[Test build #133216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133216/testReport)** for PR 30889 at commit [`6af5695`](https://github.com/apache/spark/commit/6af56953af06e5cd3d104ce38f37d23946bab6d7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-753653157


   Merged to master/3.1/3.0


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r549612115



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       I think it is possible to use `spark.read.schema` here, but need to list all the fields manually (method `schema` needs a `schema: StructType` or `schemaString: String` as input).
   So I think it maybe more complicated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bmarcott commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
bmarcott commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r549657869



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       you can create the StructType automatically from the case class like:
   `Encoders.product[NodeData].schema`
   ...but I couldn't find a way to utilize default values well, so the new column is all null.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749918109


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bmarcott commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
bmarcott commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r549657869



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       you can create the StructType automatically from the case class like:
   `Encoders.product[NodeData].schema`
   ...but I couldn't find a way to utilize default values, so the new column is all null.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749502204


   **[Test build #133216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133216/testReport)** for PR 30889 at commit [`6af5695`](https://github.com/apache/spark/commit/6af56953af06e5cd3d104ce38f37d23946bab6d7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749907054


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749926528


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37852/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749501070


   those models are created by following lines, in 2.4.7:
   ```
   
   import org.apache.spark.ml.classification._
   import org.apache.spark.ml.regression._
   
   
   val data = spark.read.format("libsvm").load("/d0/Dev/Opensource/spark/data/mllib/sample_libsvm_data.txt")
   
   
   val dtc = new DecisionTreeClassifier()
   val dtcm = dtc.fit(data)
   dtcm.save("/tmp/dtc-2.4.7")
   
   
   val rfc = new RandomForestClassifier().setNumTrees(2)
   val rfcm = rfc.fit(data)
   rfcm.save("/tmp/rfc-2.4.7")
   
   
   val gbtc = new GBTClassifier().setMaxIter(2)
   val gbtcm = gbtc.fit(data)
   gbtcm.save("/tmp/gbtc-2.4.7")
   
   
   
   
   val dtr = new DecisionTreeRegressor()
   val dtrm = dtr.fit(data)
   dtrm.save("/tmp/dtr-2.4.7")
   
   
   val rfr = new RandomForestRegressor().setNumTrees(2)
   val rfrm = rfr.fit(data)
   rfrm.save("/tmp/rfr-2.4.7")
   
   
   val gbtr = new GBTRegressor().setMaxIter(2)
   val gbtrm = gbtr.fit(data)
   gbtrm.save("/tmp/gbtr-2.4.7")
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749891510


   **[Test build #133254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133254/testReport)** for PR 30889 at commit [`027ec1c`](https://github.com/apache/spark/commit/027ec1cf9333fbc0c3435a662f8f3c418d3e7c7e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749515835


   **[Test build #133216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133216/testReport)** for PR 30889 at commit [`6af5695`](https://github.com/apache/spark/commit/6af56953af06e5cd3d104ce38f37d23946bab6d7).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749705874


   Hi, @zhengruifeng .
   I removed `[3.0][3.1]` from the PR title, `[SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0`.
   `[3.1]` is used when the PR is opened for `branch-3.1`. Please don't use it like an `Affected Version` from the JIRA issue~ Thanks. :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r596524955



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       https://github.com/apache/spark/pull/30889#issuecomment-749872161
   
   see may previous comments, and this fix come in 3.0.2




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r549164669



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       CC @zhengruifeng on this question




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Vishvin95 commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
Vishvin95 commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r596000303



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       @bmarcott @zhengruifeng Was this change applied for Spark 3.0.1 as well? I am facing the same error in Spark 3.0.1 libs. And I checked the above code inside the imported libraries and could see that the change has not been applied there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749891510


   **[Test build #133254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133254/testReport)** for PR 30889 at commit [`027ec1c`](https://github.com/apache/spark/commit/027ec1cf9333fbc0c3435a662f8f3c418d3e7c7e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749568584


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37815/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749913409


   **[Test build #133254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133254/testReport)** for PR 30889 at commit [`027ec1c`](https://github.com/apache/spark/commit/027ec1cf9333fbc0c3435a662f8f3c418d3e7c7e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749517945






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749503775


   **[Test build #133217 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133217/testReport)** for PR 30889 at commit [`8dae8fc`](https://github.com/apache/spark/commit/8dae8fce7894fc29187fbd95b9c0b9f3376a321e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bmarcott commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
bmarcott commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-753745535


   @zhengruifeng Thanks for putting in the fix!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749913620


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133254/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bmarcott commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
bmarcott commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r547927241



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       I guess spark schema doesn't allow setting default values (and converted from case class) to enable something like below?
   spark.read.schema(nodeDataSchema).parquet(dataPath).as[NodeData]




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749913620


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133254/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749880352


   Tree model will create datafrme with numPartitions=`defaultParallelism`, which will generate lots of small files; while other models like `LinearSVC` will repartition(1) before saving.
   It maybe nice to reduce the num of files in another ticket.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749539308


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37815/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Vishvin95 commented on a change in pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
Vishvin95 commented on a change in pull request #30889:
URL: https://github.com/apache/spark/pull/30889#discussion_r596000303



##########
File path: mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala
##########
@@ -401,8 +403,13 @@ private[ml] object DecisionTreeModelReadWrite {
     }
 
     val dataPath = new Path(path, "data").toString
-    val data = sparkSession.read.parquet(dataPath).as[NodeData]
-    buildTreeFromNodes(data.collect(), impurityType)
+    var df = sparkSession.read.parquet(dataPath)

Review comment:
       @bmarcott Was this change applied for Spark 3.0.1 as well? I am facing the same error in Spark 3.0.1 libs. And I checked the above code inside the imported libraries and could see that the change has not been applied there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WeichenXu123 commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-753742450


   Good works!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749568870


   So most of the large number of files are just new test files saved by 2.x? OK.
   I think it's fine to fix and add tests. this should go into 3.0.x too?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #30889: [SPARK-33398] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749904826


   also ping @imatiach-msft 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #30889: [SPARK-33398][3.0][3.1] Fix loading tree models prior to Spark 3.0

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30889:
URL: https://github.com/apache/spark/pull/30889#issuecomment-749506883


   **[Test build #133217 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133217/testReport)** for PR 30889 at commit [`8dae8fc`](https://github.com/apache/spark/commit/8dae8fce7894fc29187fbd95b9c0b9f3376a321e).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org