Reading Data from zip/gzip

[chrisr123 <ch...@gmail.com>]

I'm able to read normal txt or csv files using Flink,
but what would I need to do in order to read them if they 
are given to me in zip or gzip format? Assuming I do not want
to have to unzip them.
Thanks!




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Reading Data from zip/gzip

[Amit Jain <aj...@gmail.com>]

Hi Chris,

FileInputFormat automatically takes cares of file decompression for the
files with gzip, xz, bz2 and deflate extensions.

--
Thanks,
Amit

Source:
https://github.com/apache/flink/blob/7b040b915504e59243c642b1f4a84c956d96d134/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L118

private static void initDefaultInflaterInputStreamFactories() {
   InflaterInputStreamFactory<?>[] defaultFactories = {
         DeflateInflaterInputStreamFactory.getInstance(),
         GzipInflaterInputStreamFactory.getInstance(),
         Bzip2InputStreamFactory.getInstance(),
         XZInputStreamFactory.getInstance(),
   };
   for (InflaterInputStreamFactory<?> inputStreamFactory : defaultFactories) {
      for (String fileExtension :
inputStreamFactory.getCommonFileExtensions()) {
         registerInflaterInputStreamFactory(fileExtension, inputStreamFactory);
      }
   }
}


On Mon, Oct 22, 2018 at 2:03 PM chrisr123 <ch...@gmail.com> wrote:

> I'm able to read normal txt or csv files using Flink,
> but what would I need to do in order to read them if they
> are given to me in zip or gzip format? Assuming I do not want
> to have to unzip them.
> Thanks!
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>