[jira] [Resolved] (SPARK-22925) ml model persistence creates a lot of small files

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-22925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-22925.
----------------------------------
    Resolution: Incomplete

> ml model persistence creates a lot of small files
> -------------------------------------------------
>
>                 Key: SPARK-22925
>                 URL: https://issues.apache.org/jira/browse/SPARK-22925
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>    Affects Versions: 2.1.2, 2.2.1, 2.3.0
>            Reporter: Felix Cheung
>            Priority: Major
>              Labels: bulk-closed
>
> Today in when calling model.save(), some ML models we do makeRDD(data, 1) or repartition(1) but in some other models we don't.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60
> In the former case issues such as SPARK-19294 have been reported for making very large single file.
> Whereas in the latter case, models such as RandomForestModel could create hundreds or thousands of very small files which is also unmanageable. Looking into this, there is no simple way to set/change spark.default.parallelism (which would be pick up by sc.parallelize) while the app is running since SparkConf seems to be copied/cached by the backend without a way to update them.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135
> It seems we need to have a way to make numSlice settable on a per-use basis.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org