[jira] [Created] (IMPALA-9004) TestCompressedFormats is broken for text files

["Sahil Takiar (Jira)" <ji...@apache.org>]

Sahil Takiar created IMPALA-9004:
------------------------------------

             Summary: TestCompressedFormats is broken for text files
                 Key: IMPALA-9004
                 URL: https://issues.apache.org/jira/browse/IMPALA-9004
             Project: IMPALA
          Issue Type: Test
            Reporter: Sahil Takiar


While working on IMPALA-8950, we made a fix to {{TestCompressedFormats}} so that it actually checks the exit status of the {{hdfs dfs -cp}} command, turns out that this command has been silently failing whenever {{test_compressed_formats}} runs with {{file_format}} = {{text}}.

For some reason, data load writes compressed text files with their corresponding file compression suffix, but for compressed seq/rc files, it does not:
{code:java}
hdfs dfs -ls /test-warehouse/tinytable_seq_*
Found 1 items
-rwxr-xr-x   3 systest supergroup        325 2019-08-22 14:32 /test-warehouse/tinytable_seq_bzip/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        215 2019-08-22 14:32 /test-warehouse/tinytable_seq_def/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        260 2019-08-22 14:32 /test-warehouse/tinytable_seq_gzip/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        301 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_bzip/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        209 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_def/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        242 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_gzip/000000_0
Found 1 items
-rwxr-xr-x   3 systest supergroup        233 2019-08-22 14:32 /test-warehouse/tinytable_seq_record_snap/000000_0
Found 2 items
-rwxr-xr-x   3 systest supergroup        243 2019-08-22 14:32 /test-warehouse/tinytable_seq_snap/000000_0

hdfs dfs -ls /test-warehouse/tinytable_text_*
Found 1 items
-rwxr-xr-x   3 systest supergroup         59 2019-08-22 14:32 /test-warehouse/tinytable_text_bzip/000000_0.bz2
Found 1 items
-rwxr-xr-x   3 systest supergroup         28 2019-08-22 14:32 /test-warehouse/tinytable_text_def/000000_0.deflate
Found 1 items
-rwxr-xr-x   3 systest supergroup         40 2019-08-22 14:32 /test-warehouse/tinytable_text_gzip/000000_0.gz
Found 2 items
-rwxr-xr-x   3 systest supergroup         87 2019-08-22 14:32 /test-warehouse/tinytable_text_lzo/000000_0.lzo
-rw-r--r--   3 systest supergroup          8 2019-08-22 14:42 /test-warehouse/tinytable_text_lzo/000000_0.lzo.index
Found 1 items
-rwxr-xr-x   3 systest supergroup         41 2019-08-22 14:32 /test-warehouse/tinytable_text_snap/000000_0.snappy{code}
Not sure if that is by design or not, but it is causing the tests to fail for all text files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)