You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by Frank Grimes <fr...@gmail.com> on 2012/01/12 16:40:24 UTC

Can spill to disk be in compressed format to reduce I/O?

Hi All,

We're trying to speed up an M/R job which combines multiple .avro files.
We've noticed that when it spills to disk, it's in uncompressed format.
Is there a way to make it spill temporary segments as .avro with Deflate compression?

Thanks,

Frank Grimes