You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Srinivasan (JIRA)" <ji...@apache.org> on 2018/04/12 10:53:00 UTC
[jira] [Updated] (SPARK-23969) Using FPGrowth with PipelinedRDD
gives EOF Error
[ https://issues.apache.org/jira/browse/SPARK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Srinivasan updated SPARK-23969:
-------------------------------
Attachment: EOFError.docx
> Using FPGrowth with PipelinedRDD gives EOF Error
> ------------------------------------------------
>
> Key: SPARK-23969
> URL: https://issues.apache.org/jira/browse/SPARK-23969
> Project: Spark
> Issue Type: Bug
> Components: MLlib, Spark Submit
> Affects Versions: 2.0.2
> Environment: Spark 2.0.2
> Python2.6 (This is due to a slight issue with the Altiscale environment we are using. We will be moving to 3.5)
> Driver Memory = 32GB
> 5 nodes with 4 cores each and 32GB memory
> Shuffle Service = False
> Dynamic Allocation = False
> Cross Join = True
> Reporter: Srinivasan
> Priority: Major
> Attachments: EOFError.docx
>
>
> I am trying to find association rule on a data set with 27 million rows and 9 columns. The data is stored in a hive table and loaded into an RDD.
> I am not using the collect function as I keep getting out of memory error. I understand that FPGrowth needs a list of transactions. Which means I need to convert my dataframe into a list of lists. But I keep getting serializer.py EOF error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org