[jira] [Commented] (IMPALA-2998) impala-shell -B and --output_delimiter does not work if string contains delimiter or TABs

["David Palmer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755008#comment-16755008 ] 

David Palmer commented on IMPALA-2998:
--------------------------------------

Sorry, I'm confused.  I've experienced this behaviour with impala-shell using  impalad version 2.11.0-cdh5.14.2 and I've read through the linked issues.  Is it fixed (i.e. impala pointing to hiveserver2) in a later version, or is this something that will not be fixed in impala-shell.  It seems to be ok in Impala in Hue.

> impala-shell -B and --output_delimiter does not work if string contains delimiter or TABs
> -----------------------------------------------------------------------------------------
>
>                 Key: IMPALA-2998
>                 URL: https://issues.apache.org/jira/browse/IMPALA-2998
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Clients
>    Affects Versions: Impala 2.3.0
>            Reporter: Eric Lin
>            Priority: Major
>              Labels: impala-shell
>
> See the test case below:
> {code}
> impala-shell -q "DROP TABLE IF EXISTS tabtest";
> impala-shell -q "CREATE TABLE tabtest(col1 string,col2 string) ROW FORMAT DELIMITED FIELDS TERMINATED by ','";
> impala-shell -q 'INSERT OVERWRITE TABLE tabtest VALUES ("test", "\t\t\tTest"), ("test2", "Test\t\t\tTest"), ("test3", "test\tTest"), ("test4", "test,Test");';
> impala-shell -o out.csv -q "SELECT * FROM tabtest" --output_delimiter="," -B
> cat out.csv
> {code}
> The output looks like below:
> {code}
> test,,,,Test
> test2,Test,,,Test
> test3,test,Test
> test4,test
> {code}
> So two issues I can see here:
> 1. When strings contain TABs, all tabs will be replaced by delimiter
> 2. If string contains delimiter, the data after the delimiter is lost (see "test4"). According to doc: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_shell_options.html, 
> {quote}
> If an output value contains the delimiter character, that field is quoted and/or escaped
> {quote}
> By looking at the underlining data:
> {code}
> hadoop fs -cat /user/hive/warehouse/tabtest/ba44046cba4c7c80-d6c4c08afd8c0cb0_1055158928_data.0.
> test,			Test
> test2,Test			Test
> test3,test	Test
> test4,test,Test
> {code}
> Data is not stored properly, as they should be in quotes for those strings that contains delimter characters.
> This is both data write as well as read/parse issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org