You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Surendranauth Hiraman <su...@velos.io> on 2014/06/11 23:56:23 UTC

Compression with DISK_ONLY persistence

Hi,

Will spark.rdd.compress=true enable compression when using DISK_ONLY
persistence?



SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io

Re: Compression with DISK_ONLY persistence

Posted by Matei Zaharia <ma...@gmail.com>.

Yes, actually even if you don’t set it to true, on-disk data is compressed. (This setting only affects serialized data in memory).

Matei

On Jun 11, 2014, at 2:56 PM, Surendranauth Hiraman <su...@velos.io> wrote:

> Hi,
> 
> Will spark.rdd.compress=true enable compression when using DISK_ONLY persistence? 
> 
> 
>                                                             
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
> 
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@velos.io
> W: www.velos.io
>

Re: Compression with DISK_ONLY persistence

Posted by Matei Zaharia <ma...@gmail.com>.

Yes, actually even if you don’t set it to true, on-disk data is compressed. (This setting only affects serialized data in memory).

Matei

On Jun 11, 2014, at 2:56 PM, Surendranauth Hiraman <su...@velos.io> wrote:

> Hi,
> 
> Will spark.rdd.compress=true enable compression when using DISK_ONLY persistence? 
> 
> 
>                                                             
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
> 
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@velos.io
> W: www.velos.io
>