You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/07/15 18:25:00 UTC
[jira] [Commented] (IMPALA-10732) Use consistent DDL for specifying
Iceberg partitions
[ https://issues.apache.org/jira/browse/IMPALA-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381525#comment-17381525 ]
ASF subversion and git services commented on IMPALA-10732:
----------------------------------------------------------
Commit d0749d59de7c81a2f138986626fef12ea44e6a2c in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d0749d5 ]
IMPALA-10732: Use consistent DDL for specifying Iceberg partitions
Currently we have a DDL syntax for defining Iceberg partitions that
differs from SparkSQL:
https://iceberg.apache.org/spark-ddl/#partitioned-by
E.g. Impala is using the following syntax:
CREATE TABLE ice_t (i int, s string, ts timestamp, d date)
PARTITION BY SPEC (i BUCKET 5, ts MONTH, d YEAR)
STORED AS ICEBERG;
The same in Spark is:
CREATE TABLE ice_t (i int, s string, ts timestamp, d date)
USING ICEBERG
PARTITIONED BY (bucket(5, i), months(ts), years(d))
HIVE-25179 added the following syntax for Hive:
CREATE TABLE ice_t (i int, s string, ts timestamp, d date)
PARTITIONED BY SPEC (bucket(5, i), months(ts), years(d))
STORED BY ICEBERG;
I.e. the same syntax as Spark, but adding the keyword "SPEC".
This patch makes Impala use Hive's syntax, i.e. we will also
use the PARTITIONED BY SPEC clause + the unified partition
transform syntax.
Testing:
* existing tests has been rewritten with the new syntax
Change-Id: Ib72ae445fd68fb0ab75d87b34779dbab922bbc62
Reviewed-on: http://gerrit.cloudera.org:8080/17575
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Use consistent DDL for specifying Iceberg partitions
> ----------------------------------------------------
>
> Key: IMPALA-10732
> URL: https://issues.apache.org/jira/browse/IMPALA-10732
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Currently we have a DDL syntax for defining Iceberg partitions that differs from SparkSQL:
> [https://iceberg.apache.org/spark-ddl/#partitioned-by]
>
> E.g. Impala is using the following syntax:
>
> CREATE TABLE ice_t (i int, s string, ts timestamp, d date)
> *PARTITION BY SPEC (i BUCKET 5, ts MONTH, d YEAR)*
> STORED AS ICEBERG;
> The same in Spark is:
> CREATE TABLE ice_t (i int, s string, ts timestamp, d date)
> USING ICEBERG
> *PARTITIONED BY (bucket(5, i), months(ts), years(d))*
>
> Impala's syntax is older but hasn't been released yet. Spark's syntax is released so it cannot be changed.
>
> Hive is also working on DDL support for Iceberg partitions, and they are favoring the released SparkSQL syntax. See HIVE-25179
>
> After dicsussing it on dev@impala we decided to use SparkSQL's syntax.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org