You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Peter Vary (JIRA)" <ji...@apache.org> on 2016/09/01 22:38:20 UTC
[jira] [Issue Comment Deleted] (HIVE-14146) Column comments with
"\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Vary updated HIVE-14146:
------------------------------
Comment: was deleted
(was: It might be confusing, but here is what the patch changes:
describe escape_comments_tbl1;
{noformat:title=Beeline original}
+--------------------------+-----------------------+-----------------------+--+
| col_name | data_type | comment |
+--------------------------+-----------------------+-----------------------+--+
| col1 | string | a |
| b | NULL | NULL |
| p1 | string | a |
| b | NULL | NULL |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| p1 | string | a |
| b | NULL | NULL |
+--------------------------+-----------------------+-----------------------+--+
{noformat}
{noformat:title=Beeline patch}
+--------------------------+-----------------------+-----------------------+--+
| col_name | data_type | comment |
+--------------------------+-----------------------+-----------------------+--+
| col1 | string | a\nb |
| p1 | string | a\nb |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| p1 | string | a\nb |
+--------------------------+-----------------------+-----------------------+--+
{noformat}
{noformat:title=Cli original}
col1 string a
b
p1 string a
b
# Partition Information
# col_name data_type comment
p1 string a
b
{noformat}
{noformat:title=Cli patch}
col1 string a
b
p1 string a
b
# Partition Information
# col_name data_type comment
p1 string a
b
{noformat}
describe formatted escape_comments_tbl1;
{noformat:title=Beeline original}
+-------------------------------+----------------------------------------------------+-----------------------+--+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+-----------------------+--+
| # col_name | data_type | comment |
| | NULL | NULL |
| col1 | string | a |
| b | NULL | NULL |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| p1 | string | a |
| b | NULL | NULL |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| Owner: | petervary | NULL |
| CreateTime: | Fri Jul 01 06:40:51 PDT 2016 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1 | NULL |
| Table Type: | MANAGED_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | comment | a\nb |
| | transient_lastDdlTime | 1467380451 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-------------------------------+----------------------------------------------------+-----------------------+--+
{noformat}
{noformat:title=Beeline patch}
+-------------------------------+----------------------------------------------------+-----------------------+--+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+-----------------------+--+
| # col_name | data_type | comment |
| | NULL | NULL |
| col1 | string | a\nb |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| p1 | string | a\nb |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| Owner: | petervary | NULL |
| CreateTime: | Fri Jul 01 06:40:51 PDT 2016 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1 | NULL |
| Table Type: | MANAGED_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | comment | a\nb |
| | transient_lastDdlTime | 1467380451 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-------------------------------+----------------------------------------------------+-----------------------+--+
{noformat}
{noformat:title=Cli original}
# col_name data_type comment
col1 string a
b
# col_name data_type comment
p1 string a
b
# Detailed Table Information
Database: default
Owner: petervary
CreateTime: Fri Jul 01 06:40:51 PDT 2016
LastAccessTime: UNKNOWN
Retention: 0
Location: file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1
Table Type: MANAGED_TABLE
Table Parameters:
comment a\nb
transient_lastDdlTime 1467380451
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
{noformat}
{noformat:title=Cli patch}
# col_name data_type comment
col1 string a
b
# Partition Information
# col_name data_type comment
p1 string a
b
# Detailed Table Information
Database: default
Owner: petervary
CreateTime: Fri Jul 01 06:40:51 PDT 2016
LastAccessTime: UNKNOWN
Retention: 0
Location: file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1
Table Type: MANAGED_TABLE
Table Parameters:
comment a
b
transient_lastDdlTime 1467380451
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
{noformat}
describe pretty escape_comments_tbl1;
{noformat:title=Beeline original}
+--------------------------+-----------------+----------+--+
| col_name | data_type | comment |
+--------------------------+-----------------+----------+--+
| col_name | data_type | comment |
| | | |
| col1 | string | a |
| | | b |
| p1 | string | a |
| | | b |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| col_name | data_type | comment |
| | | |
| p1 | string | a |
| | | b |
+--------------------------+-----------------+----------+--+
{noformat}
{noformat:title=Beeline patch}
+--------------------------+-----------------+----------+--+
| col_name | data_type | comment |
+--------------------------+-----------------+----------+--+
| col_name | data_type | comment |
| | | |
| col1 | string | a |
| | | b |
| p1 | string | a |
| | | b |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| col_name | data_type | comment |
| | | |
| p1 | string | a |
| | | b |
+--------------------------+-----------------+----------+--+
{noformat}
{noformat:title=Cli original}
col_name data_type comment
col1 string a
b
p1 string a
b
# Partition Information
col_name data_type comment
p1 string a
b
{noformat}
{noformat:title=Cli patch}
col_name data_type comment
col1 string a
b
p1 string a
b
# Partition Information
col_name data_type comment
p1 string a
b
{noformat}
describe database escape_comments_db;
{noformat:title=Beeline original}
+---------------------+----------------------------------------------------+------------+-------------+-------------+-------------+--+
2 rows selected (0.054 seconds)
| db_name | comment | location | owner_name | owner_type | parameters |
+---------------------+----------------------------------------------------+------------+-------------+-------------+-------------+--+
| escape_comments_db | a | NULL | NULL | NULL | NULL |
| b | file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db | petervary | USER | | NULL |
+---------------------+----------------------------------------------------+------------+-------------+-------------+-------------+--+
{noformat}
{noformat:title=Beeline patch}
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
| db_name | comment | location | owner_name | owner_type | parameters |
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
| escape_comments_db | a\nb | file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db | petervary | USER | |
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
{noformat}
{noformat:title=Cli original}
escape_comments_db a
b file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db petervary USER
{noformat}
{noformat:title=Cli patch}
escape_comments_db a\nb file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db petervary USER
{noformat}
describe database extended escape_comments_db;
{noformat:title=Beeline original}
escape_comments_db a
b file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db petervary USER
{noformat}
{noformat:title=Beeline patch}
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
| db_name | comment | location | owner_name | owner_type | parameters |
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
| escape_comments_db | a\nb | file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db | petervary | USER | |
+---------------------+----------+----------------------------------------------------+-------------+-------------+-------------+--+
{noformat}
{noformat:title=Cli original}
escape_comments_db a
b file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db petervary USER
{noformat}
{noformat:title=Cli patch}
escape_comments_db a\nb file:/Users/petervary/data/apache/hive/warehouse/escape_comments_db.db petervary USER
{noformat}
describe formatted escape_comments_view1;
{noformat:title=Beeline original}
+-------------------------------+----------------------------------------------------+-----------------------+--+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+-----------------------+--+
| # col_name | data_type | comment |
| | NULL | NULL |
| col1 | string | a |
| b | NULL | NULL |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| Owner: | petervary | NULL |
| CreateTime: | Fri Jul 01 06:40:51 PDT 2016 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Table Type: | VIRTUAL_VIEW | NULL |
| Table Parameters: | NULL | NULL |
| | comment | a\nb |
| | transient_lastDdlTime | 1467380451 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | null | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| | NULL | NULL |
| # View Information | NULL | NULL |
| View Original Text: | select col1 from escape_comments_tbl1 | NULL |
| View Expanded Text: | SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1` | NULL |
+-------------------------------+----------------------------------------------------+-----------------------+--+
{noformat}
{noformat:title=Beeline patch}
+-------------------------------+----------------------------------------------------+-----------------------+--+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+-----------------------+--+
| # col_name | data_type | comment |
| | NULL | NULL |
| col1 | string | a\nb |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| Owner: | petervary | NULL |
| CreateTime: | Fri Jul 01 06:40:51 PDT 2016 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Table Type: | VIRTUAL_VIEW | NULL |
| Table Parameters: | NULL | NULL |
| | comment | a\nb |
| | transient_lastDdlTime | 1467380451 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | null | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| | NULL | NULL |
| # View Information | NULL | NULL |
| View Original Text: | select col1 from escape_comments_tbl1 | NULL |
| View Expanded Text: | SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1` | NULL |
+-------------------------------+----------------------------------------------------+-----------------------+--+
{noformat}
{noformat:title=Cli original}
# col_name data_type comment
col1 string a
b
# Detailed Table Information
Database: default
Owner: petervary
CreateTime: Fri Jul 01 06:40:51 PDT 2016
LastAccessTime: UNKNOWN
Retention: 0
Table Type: VIRTUAL_VIEW
Table Parameters:
comment a\nb
transient_lastDdlTime 1467380451
# Storage Information
SerDe Library: null
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
# View Information
View Original Text: select col1 from escape_comments_tbl1
View Expanded Text: SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1`
{noformat}
{noformat:title=Cli patch}
# col_name data_type comment
col1 string a
b
# Detailed Table Information
Database: default
Owner: petervary
CreateTime: Fri Jul 01 06:40:51 PDT 2016
LastAccessTime: UNKNOWN
Retention: 0
Table Type: VIRTUAL_VIEW
Table Parameters:
comment a
b
transient_lastDdlTime 1467380451
# Storage Information
SerDe Library: null
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
# View Information
View Original Text: select col1 from escape_comments_tbl1
View Expanded Text: SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1`
{noformat}
show create table escape_comments_tbl1;
{noformat:title=Beeline original}
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE TABLE `escape_comments_tbl1`( |
| `col1` string COMMENT 'a |
| b') |
| COMMENT 'a |
| b' |
| PARTITIONED BY ( |
| `p1` string COMMENT 'a |
| b') |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1' |
| TBLPROPERTIES ( |
| 'transient_lastDdlTime'='1467380451') |
+----------------------------------------------------+--+
{noformat}
{noformat:title=Beeline patch}
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE TABLE `escape_comments_tbl1`( |
| `col1` string COMMENT 'a\nb') |
| COMMENT 'a\nb' |
| PARTITIONED BY ( |
| `p1` string COMMENT 'a\nb') |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1' |
| TBLPROPERTIES ( |
| 'transient_lastDdlTime'='1467380451') |
+----------------------------------------------------+--+
{noformat}
{noformat:title=Cli original}
CREATE TABLE `escape_comments_tbl1`(
`col1` string COMMENT 'a
b')
COMMENT 'a
b'
PARTITIONED BY (
`p1` string COMMENT 'a
b')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1'
TBLPROPERTIES (
'transient_lastDdlTime'='1467380451')
{noformat}
{noformat:title=Cli patch}
CREATE TABLE `escape_comments_tbl1`(
`col1` string COMMENT 'a\nb')
COMMENT 'a\nb'
PARTITIONED BY (
`p1` string COMMENT 'a\nb')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/Users/petervary/data/apache/hive/warehouse/escape_comments_tbl1'
TBLPROPERTIES (
'transient_lastDdlTime'='1467380451')
{noformat}
show create table escape_comments_view1;
{noformat:title=Beeline original}
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE VIEW `escape_comments_view1` AS SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1` |
+----------------------------------------------------+--+
{noformat}
{noformat:title=Beeline patch}
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE VIEW `escape_comments_view1` AS SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1` |
+----------------------------------------------------+--+
{noformat}
{noformat:title=Cli original}
CREATE VIEW `escape_comments_view1` AS SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1`
{noformat}
{noformat:title=Cli patch}
CREATE VIEW `escape_comments_view1` AS SELECT `col1` AS `col1` FROM (select `escape_comments_tbl1`.`col1` from `default`.`escape_comments_tbl1`) `default.escape_comments_view1`
{noformat}
show formatted index on escape_comments_tbl1;
{noformat:title=Beeline original}
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
| idx_name | tab_name | col_names | idx_tab_name | idx_type | comment |
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
| idx_name | tab_name | col_names | idx_tab_name | idx_type | comment |
| | NULL | NULL | NULL | NULL | NULL |
| | NULL | NULL | NULL | NULL | NULL |
| index2 | escape_comments_tbl1 | col1 | default__escape_comments_tbl1_index2__ | compact | a |
| b | | NULL | NULL | NULL | NULL |
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
{noformat}
{noformat:title=Beeline patch}
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
| idx_name | tab_name | col_names | idx_tab_name | idx_type | comment |
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
| idx_name | tab_name | col_names | idx_tab_name | idx_type | comment |
| | NULL | NULL | NULL | NULL | NULL |
| | NULL | NULL | NULL | NULL | NULL |
| index2 | escape_comments_tbl1 | col1 | default__escape_comments_tbl1_index2__ | compact | a\nb |
+-----------------------+-----------------------+-----------------------+-----------------------------------------+-----------------------+-----------------------+--+
{noformat}
{noformat:title=Cli original}
idx_name tab_name col_names idx_tab_name idx_type comment
index2 escape_comments_tbl1 col1 default__escape_comments_tbl1_index2__ compact a
b
{noformat}
{noformat:title=Cli patch}
idx_name tab_name col_names idx_tab_name idx_type comment
index2 escape_comments_tbl1 col1 default__escape_comments_tbl1_index2__ compact a
b
{noformat})
> Column comments with "\n" character "corrupts" table metadata
> -------------------------------------------------------------
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
> Affects Versions: 2.2.0
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-14146.10.patch, HIVE-14146.2.patch, HIVE-14146.3.patch, HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch, HIVE-14146.7.patch, HIVE-14146.8.patch, HIVE-14146.9.patch, HIVE-14146.patch, changes
>
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +-------------------+------------+-----------------------+--+
> | col_name | data_type | comment |
> +-------------------+------------+-----------------------+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +-------------------+------------+-----------------------+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)