[jira] [Created] (BEAM-167) TextIO can't read concatenated gzip files

["Eugene Kirpichov (JIRA)" <ji...@apache.org>]

Eugene Kirpichov created BEAM-167:
-------------------------------------

             Summary: TextIO can't read concatenated gzip files
                 Key: BEAM-167
                 URL: https://issues.apache.org/jira/browse/BEAM-167
             Project: Beam
          Issue Type: Bug
            Reporter: Eugene Kirpichov


$ cat <<END > header.csv
a,b,c
END
$ cat <<END > body.csv
1,2,3
4,5,6
7,8,9
END
$ gzip -c header.csv > file.gz
$ gzip -c body.csv >> file.gz

The file is well-formed:
$ gzip -dc file.gz
a,b,c
1,2,3
4,5,6
7,8,9

However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - reproducible even when the file is on local disk and with the DirectPipelineRunner.

The bug is in CompressedSource. It uses GzipCompressorInputStream, which by default reads only the first gzip stream in the file, but has an option to read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)