You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2015/08/21 15:57:56 UTC

Want to install lz4 compression

Hi all,

I am using pre-compiled spark with hadoop 2.6. LZ4 Codec is not on hadoop's native libraries, so I am not being able to use it.

Can anyone suggest on how to proceed? Hopefully I wont have to recompile hadoop. I tried changing the --driver-library-path to point directly into lz4 stand alone package libraries, but of course it didnt work.

Thanks
Saif

Re: Want to install lz4 compression

Posted by Ted Yu <yu...@gmail.com>.

Have you read this ?

http://stackoverflow.com/questions/22716346/how-to-use-lz4-compression-in-linux-3-11



> On Aug 21, 2015, at 6:57 AM, <Sa...@wellsfargo.com> <Sa...@wellsfargo.com> wrote:
> 
> Hi all,
>  
> I am using pre-compiled spark with hadoop 2.6. LZ4 Codec is not on hadoop’s native libraries, so I am not being able to use it.
>  
> Can anyone suggest on how to proceed? Hopefully I wont have to recompile hadoop. I tried changing the --driver-library-path to point directly into lz4 stand alone package libraries, but of course it didnt work.
>  
> Thanks
> Saif
>