[Spark R]could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

[Todd <bi...@163.com>]

I am launching spark R with following script:

./sparkR --driver-memory 12G

and I try to load a local 3G csv file with following code,

> a=read.transactions("/home/admin/datamining/data.csv",sep="\t",format="single",cols=c(1,2))

but I encounter an error:  could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

I have allocated 12G memory,not sure why it complains could not allocate 2G memory.

Could someone help me? Thanks!

RE: [Spark R]could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

["Sun, Rui" <ru...@intel.com>]

Hi,Todd,

"--driver-memory" options specifies the maximum heap memory size of the JVM backend for SparkR. The error you faced is memory allocation error of your R process. They are different.
I guess that 2G memory bound for a string is limitation of the R interpreter?
That's the reason why we use SparkR for big data processing. Maybe you can try copy your CSV to HDFS and have it partitioned across a cluster. But I am not sure if Spark can support distributed CSV file or not.

From: Todd [mailto:bit1129@163.com]
Sent: Friday, November 6, 2015 5:00 PM
To: user@spark.apache.org
Subject: [Spark R]could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

I am launching spark R with following script:

./sparkR --driver-memory 12G

and I try to load a local 3G csv file with following code,

> a=read.transactions("/home/admin/datamining/data.csv",sep="\t",format="single",cols=c(1,2))

but I encounter an error:  could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

I have allocated 12G memory,not sure why it complains could not allocate 2G memory.

Could someone help me? Thanks!