[jira] [Resolved] (SPARK-6240) Spark MLlib fpm#FPGrowth genFreqItems use Array[Item] may outOfMemory for Large Sets

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-6240.
------------------------------
    Resolution: Not a Problem

(This would be better to start as a question on the mailing list.) 

It sounds like you have constructed your test case so that there are a billion "frequent" items. The general idea of this algorithm is that there are relatively few frequent items. This will still work, but you'll need several gigabytes of memory on the driver and on your workers; that is, you're simply running out of memory for what you are asking to do, so one answer is to increase your memory from the default of 512M / 1G.

>From what I know of FP-growth, I don't think it's realistic to make the initial header table of frequent items a distributed structure. It would complicate and greatly slow down the common case where the min support produces far fewer frequent items.

> Spark MLlib fpm#FPGrowth genFreqItems use Array[Item] may outOfMemory for Large Sets
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-6240
>                 URL: https://issues.apache.org/jira/browse/SPARK-6240
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>    Affects Versions: 1.3.0
>            Reporter: Littlestar
>            Priority: Minor
>
> Spark MLlib fpm#FPGrowth genFreqItems use Array[Item] may outOfMemory for Large Sets
> {noformat}
>   private def genFreqItems[Item: ClassTag](
>       data: RDD[Array[Item]],
>       minCount: Long,
>       partitioner: Partitioner): Array[Item] = {
>     data.flatMap { t =>
>       val uniq = t.toSet
>       if (t.size != uniq.size) {
>         throw new SparkException(s"Items in a transaction must be unique but got ${t.toSeq}.")
>       }
>       t
>     }.map(v => (v, 1L))
>       .reduceByKey(partitioner, _ + _)
>       .filter(_._2 >= minCount)
>       .collect()
>       .sortBy(-_._2)
>       .map(_._1)
>   }
> {noformat}
> I use 10*10000*10000 records for test, for output all simultaneously pair.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org