Any limitations of spark.shuffle.spill?

[Yangcheng Huang <ya...@huawei.com>]

Hi

One question about the power of spark.shuffle.spill -
(I know this has been asked several times :-)

Basically, in handling a (cached) dataset that doesn't fit in memory, Spark can spill it to disk.

However, can I say that, when this is enabled, Spark can handle the situation faultlessly, no matter -

(1)    How big the data set is (as compared to the available memory)

(2)    How complex the detailed calculation is being carried out
Can spark.shuffle.spill handle this perfectly?

Here we assume that (1) the disk space has no limitations and (2) the code is correctly written according to the functional requirements.

The reason to ask this is, under such situations, I kept receiving warnings like "FetchFailed", if memory usage reaches the limit.

Thanks
YC

Re: Any limitations of spark.shuffle.spill?

[Shixiong Zhu <zs...@gmail.com>]

Two limitations we found here:
http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemory-in-quot-cogroup-quot-td17349.html

Best Regards,
Shixiong Zhu

2014-11-06 2:04 GMT+08:00 Yangcheng Huang <ya...@huawei.com>:

>  Hi
>
>
>
> One question about the power of spark.shuffle.spill –
>
> (I know this has been asked several times :-)
>
>
>
> Basically, in handling a (cached) dataset that doesn’t fit in memory,
> Spark can spill it to disk.
>
>
>
> However, can I say that, when this is enabled, Spark can handle the
> situation faultlessly, no matter –
>
> (1)    How big the data set is (as compared to the available memory)
>
> (2)    How complex the detailed calculation is being carried out
>
> Can spark.shuffle.spill handle this perfectly?
>
>
>
> Here we assume that (1) the disk space has no limitations and (2) the code
> is correctly written according to the functional requirements.
>
>
>
> The reason to ask this is, under such situations, I kept receiving
> warnings like “FetchFailed”, if memory usage reaches the limit.
>
>
>
> Thanks
>
> YC
>