Re: zstd codec

["Driesprong, Fokko" <fo...@driesprong.frl>]

Yes, it should be part of your Hadoop distribution. Since the zstd will
call native implementations to speed up the decompression these libraries
need to be there at compile-time to wire everything correctly.

 You can check if they are:

 $ hadoop checknative -a
   14/12/06 01:30:45 WARN bzip2.Bzip2Factory: Failed to
load/initialize native-bzip2 library system-native, will use pure-Java
version
   14/12/06 01:30:45 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
   Native library checking:
   hadoop: true /home/ozawa/hadoop/lib/native/libhadoop.so.1.0.0
   zlib:   true /lib/x86_64-linux-gnu/libz.so.1
   snappy: true /usr/lib/libsnappy.so.1
   zstd: true /usr/lib/libzstd.so.1
   lz4:    true revision:99
   bzip2:  false


Hope this helps.


Cheers, Fokko




Op wo 31 jul. 2019 om 19:03 schreef Manik Singla <sm...@gmail.com>:

> I downloaded and added libHadoop.so to make it work.
> I will compile it separately for mac. Lemme know if we can make it work
> using just Jars or readymade soln
>
>
> Regards
> Manik Singla
> +91-9996008893
> +91-9665639677
>
> "Life doesn't consist in holding good cards but playing those you hold
> well."
>
>
> On Wed, Jul 31, 2019 at 3:34 PM Manik Singla <sm...@gmail.com> wrote:
>
> > Hadoop common fails with native zStandard library not available: this
> > version of libhadoop was built without zstd support.
> >
> > I don't want to compile libhadoop for myself
> >
> > On Wed, 31 Jul, 2019, 3:19 PM Driesprong, Fokko, <fo...@driesprong.frl>
> > wrote:
> >
> >> Hi Manik,
> >>
> >> Can you explain how you en- and decode the messages? Please keep in mind
> >> that with Parquet only the actual data is compressed. For example, the
> >> metadata in the footer isn't compressed.
> >>
> >> Cheers, Fokko
> >>
> >> Op wo 31 jul. 2019 om 11:33 schreef Manik Singla <sm...@gmail.com>:
> >>
> >> > Hi Guys
> >> >
> >> > We are trying to use zstd but there are compatibility issues with
> >> standard
> >> > binaries.
> >> > I am using latest of parquet jars.
> >> >
> >> > Am i missing something?
> >> >
> >> > Regards
> >> > Manik Singla
> >> > +91-9996008893
> >> > +91-9665639677
> >> >
> >> > "Life doesn't consist in holding good cards but playing those you hold
> >> > well."
> >> >
> >>
> >
>