You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiaoye Sun (JIRA)" <ji...@apache.org> on 2016/12/06 01:31:58 UTC

[jira] [Created] (SPARK-18731) Task size in K-means is so large

Xiaoye Sun created SPARK-18731:
----------------------------------

             Summary: Task size in K-means is so large
                 Key: SPARK-18731
                 URL: https://issues.apache.org/jira/browse/SPARK-18731
             Project: Spark
          Issue Type: Improvement
          Components: MLlib
    Affects Versions: 1.6.1
            Reporter: Xiaoye Sun
            Priority: Minor


When run the KMeans algorithm for a large model (e.g. 100k features and 100 centers), there will be warning shown for many of the stages saying that the task size is very large. Here is an example warning. 
WARN TaskSetManager: Stage 23 contains a task of very large size (56256 KB). The maximum recommended task size is 100 KB.

This could happen at (sum at KMeansModel.scala:88), (takeSample at KMeans.scala:378), (aggregate at KMeans.scala:404) and (collect at KMeans.scala:436). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org