[jira] [Created] (IMPALA-7056) Changing Text Delimiter Does Not Work

["Alan Jackoway (JIRA)" <ji...@apache.org>]

Alan Jackoway created IMPALA-7056:
-------------------------------------

             Summary: Changing Text Delimiter Does Not Work
                 Key: IMPALA-7056
                 URL: https://issues.apache.org/jira/browse/IMPALA-7056
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog, Docs
    Affects Versions: Impala 2.12.0
            Reporter: Alan Jackoway


The wording on https://impala.apache.org/docs/build/html/topics/impala_alter_table.html makes it seem like you can change the delimiter of text tables after they are created.

I did the following to simulate a table that needed to switch between comma and pipe delimited:
{code}
hadoop fs -mkdir /user/alanj
hadoop fs -mkdir /user/alanj/test_delim
echo "A,B|C" > delim.txt
hadoop fs -put delim.txt /user/alanj/test_delim
{code}

Then created in impala and tried to change delimiters:
{code:sql}
> create external table default.alanj_test_delim(A string, B string) ROW FORMAT DELIMITED FIELDS TERMINATED BY "," LOCATION '/user/alanj/test_delim';
> select * from default.alanj_test_delim;
Query: select * from default.alanj_test_delim
+---+-----+
| a | b   |
+---+-----+
| A | B|C |
+---+-----+
> alter table default.alanj_test_delim set SERDEPROPERTIES ('serialization.format'='|', 'field.delim'='|');
> select * from default.alanj_test_delim;
+---+-----+
| a | b   |
+---+-----+
| A | B|C |
+---+-----+
> show create table default.alanj_test_delim;
+----------------------------------------------------------------------------------------------------------------------+
| result                                                                                                               |
+----------------------------------------------------------------------------------------------------------------------+
| CREATE EXTERNAL TABLE default.alanj_test_delim (                                                                     |
|   a STRING,                                                                                                          |
|   b STRING                                                                                                           |
| )                                                                                                                    |
| ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'                                                                        |
| WITH SERDEPROPERTIES ('field.delim'='|', 'serialization.format'='|')                                                 |
| STORED AS TEXTFILE                                                                                                   |
| LOCATION 'hdfs://namenode:8020/user/alanj/test_delim'                                  |
| TBLPROPERTIES ('COLUMN_STATS_ACCURATE'='false', 'numFiles'='0', 'numRows'='-1', 'rawDataSize'='-1', 'totalSize'='0') |
+----------------------------------------------------------------------------------------------------------------------+
{code}

So it shows the right serdeproperties, but impala doesn't actually use them to read the data.

If you then insert data (as the docs suggest), it writes that data with the new delimiter:
{code:sql}
> insert into default.alanj_test_delim values('D', 'E,F');
> select * from alanj_test_delim;
+-----+-----+
| a   | b   |
+-----+-----+
| A,B | C   |
| D   | E,F |
+-----+-----+
# hadoop fs -cat /user/alanj/test_delim/a54bb0ec14646492-a738811400000000_1498283208_data.0.
D|E,F
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)