You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Miklos Gergely (JIRA)" <ji...@apache.org> on 2019/06/19 23:41:00 UTC
[jira] [Comment Edited] (HIVE-21897) Setting serde / serde
properties for partitions
[ https://issues.apache.org/jira/browse/HIVE-21897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868104#comment-16868104 ]
Miklos Gergely edited comment on HIVE-21897 at 6/19/19 11:40 PM:
-----------------------------------------------------------------
[~mithun] after executing those commands:
SHOW EXTENDED foobar;
{code:java}
+-----------------------------+----------------------------------------------------+----------+
| col_name | data_type | comment |
+-----------------------------+----------------------------------------------------+----------+
| foo | string | |
| bar | string | |
| dt | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| dt | string | |
| | NULL | NULL |
| Detailed Table Information | Table(tableName:foobar, dbName:default, owner:hive, createTime:1560986681, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:foo, type:string, comment:null), FieldSchema(name:bar, type:string, comment:null), FieldSchema(name:dt, type:string, comment:null)], location:hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:dt, type:string, comment:null)], parameters:{last_modified_time=1560986885, totalSize=0, numRows=0, rawDataSize=0, transactional_properties=insert_only, COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\"}, numFiles=0, numPartitions=2, transient_lastDdlTime=1560986885, bucketing_version=2, last_modified_by=hive, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, rewriteEnabled:false, catName:hive, ownerType:USER, writeId:0) | |
{code}
SHOW CREATE TABLE foobar;
{code:java}
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE TABLE `foobar`( |
| `foo` string, |
| `bar` string) |
| PARTITIONED BY ( |
| `dt` string) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION |
| 'hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'last_modified_by'='hive', |
| 'last_modified_time'='1560986885', |
| 'transactional'='true', |
| 'transactional_properties'='insert_only', |
| 'transient_lastDdlTime'='1560986885') |
+----------------------------------------------------+
{code}
So as it seems the table has only one SerDe, not per partition. Do we want to allow a different SerDe per partition? Because if we do, it needs planning, and code changes. Or for now we may stick to the one SerDe / table.
was (Author: mgergely):
[~mithun] after executing those commands:
SHOW EXTENDED foobar;
{code:java}
+-----------------------------+----------------------------------------------------+----------+
| col_name | data_type | comment |
+-----------------------------+----------------------------------------------------+----------+
| foo | string | |
| bar | string | |
| dt | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| dt | string | |
| | NULL | NULL |
| Detailed Table Information | Table(tableName:foobar, dbName:default, owner:hive, createTime:1560986681, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:foo, type:string, comment:null), FieldSchema(name:bar, type:string, comment:null), FieldSchema(name:dt, type:string, comment:null)], location:hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:dt, type:string, comment:null)], parameters:{last_modified_time=1560986885, totalSize=0, numRows=0, rawDataSize=0, transactional_properties=insert_only, COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\"}, numFiles=0, numPartitions=2, transient_lastDdlTime=1560986885, bucketing_version=2, last_modified_by=hive, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, rewriteEnabled:false, catName:hive, ownerType:USER, writeId:0) | |
{code}
SHOW CREATE TABLE foobar;
{code:java}
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE TABLE `foobar`( |
| `foo` string, |
| `bar` string) |
| PARTITIONED BY ( |
| `dt` string) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION |
| 'hdfs://hive-on-tezt-1.vpc.cloudera.com:8020/warehouse/tablespace/managed/hive/foobar' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'last_modified_by'='hive', |
| 'last_modified_time'='1560986885', |
| 'transactional'='true', |
| 'transactional_properties'='insert_only', |
| 'transient_lastDdlTime'='1560986885') |
+----------------------------------------------------+
{code}
So as it seems the table has only one SerDe, not per partition. Do we want to allow a different SerDe per partition? Because if we do, it needs planning, and code changes. Or for now we may stick to the one SerDe / table.
> Setting serde / serde properties for partitions
> -----------------------------------------------
>
> Key: HIVE-21897
> URL: https://issues.apache.org/jira/browse/HIVE-21897
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.1
> Reporter: Miklos Gergely
> Assignee: Ashutosh Chauhan
> Priority: Major
> Fix For: 4.0.0
>
>
> According to [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AddSerDeProperties] the SerDe and the SerDe properties can be set for a partition too, so
>
> {code:java}
> ALTERT TABLE table PARTITION (partition_col='partition_value') SET SERDE 'serde.class.name';{code}
> Is a valid statement. In fact it is not rejected, but it is not doing anything at all. The execution is successful, everything remains the same. The same is true for setting the serde properties:
> {code:java}
> ALTER TABLE table PARTITION (partition_col='partition_value') SET SERDEPROPERTIES ('property_name'='property_value');{code}
> is also a valid statement, and not doing anything.
> I suggest to modify the parser, and reject these statements. SerDe is for a table, and not for a partition.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)