You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by XIANDI <zx...@hotmail.com> on 2015/10/08 21:56:06 UTC

ValueError: can not serialize object larger than 2G

  File "/home/hadoop/spark/python/pyspark/worker.py", line 101, in main
    process()
  File "/home/hadoop/spark/python/pyspark/worker.py", line 96, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/home/hadoop/spark/python/pyspark/serializers.py", line 126, in
dump_stream
    self._write_with_length(obj, stream)
  File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
_write_with_length
    raise ValueError("can not serialize object larger than 2G")
ValueError: can not serialize object larger than 2G

Does anyone know how does this happen?

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ValueError-can-not-serialize-object-larger-than-2G-tp24984.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


FW: ValueError: can not serialize object larger than 2G

Posted by Xiandi Zhang <zx...@hotmail.com>.
------------------------------------------------------------------------------------------------------------------------------------------def parse_record(x):    formatted = list(x[0])    if (type(x[1])!=list) & (type(x[1])!=tuple):        formatted.append(x[1])    else:        formatted.extend(list(x[1]))    return formatted
def writeRecords(records):    output = StringIO.StringIO()    for record in records:        writer = csv.writer(output, delimiter=',')        writer.writerow(record)    return [output.getvalue()]
validation_formatted = validation_res.map(lambda x: parse_record(x))validation_partition = validation_formatted.repartition(80)validation_partition.mapPartitions(writeRecords).saveAsTextFile(broadcast_path + "para_predictions")------------------------------------------------------------------------------------------------------------------------------------------
The error happens when saveAsTextFile. I changed the  repartition number to 200 or 300. But still got this error. 
Thanks for help.
Date: Thu, 8 Oct 2015 13:55:32 -0700
Subject: Re: ValueError: can not serialize object larger than 2G
From: yuzhihong@gmail.com
To: zxd_cindy@hotmail.com
CC: user@spark.apache.org

To fix the problem, consider increasing number of partitions for your job.
Showing code snippet would help us understand your use case better.
Cheers
On Thu, Oct 8, 2015 at 1:39 PM, Ted Yu <yu...@gmail.com> wrote:
See the comment of FramedSerializer() in serializers.py :
    Serializer that writes objects as a stream of (length, data) pairs,    where C{length} is a 32-bit integer and data is C{length} bytes.
Hence the limit on the size of object.
On Thu, Oct 8, 2015 at 12:56 PM, XIANDI <zx...@hotmail.com> wrote:
  File "/home/hadoop/spark/python/pyspark/worker.py", line 101, in main

    process()

  File "/home/hadoop/spark/python/pyspark/worker.py", line 96, in process

    serializer.dump_stream(func(split_index, iterator), outfile)

  File "/home/hadoop/spark/python/pyspark/serializers.py", line 126, in

dump_stream

    self._write_with_length(obj, stream)

  File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in

_write_with_length

    raise ValueError("can not serialize object larger than 2G")

ValueError: can not serialize object larger than 2G



Does anyone know how does this happen?



Thanks!







--

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ValueError-can-not-serialize-object-larger-than-2G-tp24984.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org






 		 	   		   		 	   		  

Re: ValueError: can not serialize object larger than 2G

Posted by Ted Yu <yu...@gmail.com>.
To fix the problem, consider increasing number of partitions for your job.

Showing code snippet would help us understand your use case better.

Cheers

On Thu, Oct 8, 2015 at 1:39 PM, Ted Yu <yu...@gmail.com> wrote:

> See the comment of FramedSerializer() in serializers.py :
>
>     Serializer that writes objects as a stream of (length, data) pairs,
>     where C{length} is a 32-bit integer and data is C{length} bytes.
>
> Hence the limit on the size of object.
>
> On Thu, Oct 8, 2015 at 12:56 PM, XIANDI <zx...@hotmail.com> wrote:
>
>>   File "/home/hadoop/spark/python/pyspark/worker.py", line 101, in main
>>     process()
>>   File "/home/hadoop/spark/python/pyspark/worker.py", line 96, in process
>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>   File "/home/hadoop/spark/python/pyspark/serializers.py", line 126, in
>> dump_stream
>>     self._write_with_length(obj, stream)
>>   File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
>> _write_with_length
>>     raise ValueError("can not serialize object larger than 2G")
>> ValueError: can not serialize object larger than 2G
>>
>> Does anyone know how does this happen?
>>
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/ValueError-can-not-serialize-object-larger-than-2G-tp24984.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Re: ValueError: can not serialize object larger than 2G

Posted by Ted Yu <yu...@gmail.com>.
See the comment of FramedSerializer() in serializers.py :

    Serializer that writes objects as a stream of (length, data) pairs,
    where C{length} is a 32-bit integer and data is C{length} bytes.

Hence the limit on the size of object.

On Thu, Oct 8, 2015 at 12:56 PM, XIANDI <zx...@hotmail.com> wrote:

>   File "/home/hadoop/spark/python/pyspark/worker.py", line 101, in main
>     process()
>   File "/home/hadoop/spark/python/pyspark/worker.py", line 96, in process
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/home/hadoop/spark/python/pyspark/serializers.py", line 126, in
> dump_stream
>     self._write_with_length(obj, stream)
>   File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
> _write_with_length
>     raise ValueError("can not serialize object larger than 2G")
> ValueError: can not serialize object larger than 2G
>
> Does anyone know how does this happen?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/ValueError-can-not-serialize-object-larger-than-2G-tp24984.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>