You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by 1 <li...@126.com> on 2021/05/10 08:07:15 UTC
(Hive) insert overwrite table xxx partition (pt='xxx') on iceberg
table
Hi, team
When I migrate tables from hive to iceberg use spark3, partition info in ddl is hidden.
When I run insert overwrite table xxx partition (pt='xxx’) on flink or spark sql-shell, it’s ok, but when I run it on hive sql-shell, I get a error like below:
So what can I do for it, specify the partition in ddl ?
| |
liubo07199
|
|
|
Re: (Hive) insert overwrite table xxx partition (pt='xxx') on iceberg table
Posted by Marton Bod <ma...@gmail.com>.
Hi,
Thanks for reaching out! Unfortunately insert overwrite commands are not
currently supported by the Hive engine in upstream Iceberg, since it's a
feature difficult to implement without touching the Hive codebase as well.
Currently we only have support for regular insert queries (using the mr
engine, not tez).
We have recently created a new module in upstream Hive called
"hive-iceberg-handler", which is supposed to be an alternative over time to
using the iceberg-hive-runtime jar, providing an extended feature set and
performance improvements. Support for insert overwrites has already been
implemented there. Please note the module is still experimental and some of
these newer features would require you to run Hive from the master branch
due to core Hive API changes, but it might be a good candidate for
experimentation. Here's the link for the module if you'd like to check out
our ongoing work:
https://github.com/apache/hive/tree/master/iceberg/iceberg-handler
As for the specific error message you're getting, it's because the HMS by
design sees the Iceberg table as unpartitioned (to enable flexible
partitioning down the line and due to how the Hive query planner works),
even though the underlying Iceberg table is actually partitioned - hence
the upstream error during the compilation phase.
Best,
Marton
On Mon, 10 May 2021 at 10:11, 1 <li...@126.com> wrote:
> Error pic cannot upload, hive error is
>
> FAILED: ValidationFailureSemanticException table is not partitioned but
> partition spec exists: {pt=xxx}
>
> liubo07199
> liubo07199@hellobike.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=liubo07199&uid=liubo07199%40hellobike.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22liubo07199%40hellobike.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 05/10/2021 16:07，1<li...@126.com> <li...@126.com> wrote：
>
> Hi, team
>
> When I migrate tables from hive to iceberg use spark3, partition info
> in ddl is hidden.
>
> When I run *insert overwrite table xxx partition (pt='xxx’)* on
> flink or spark sql-shell, it’s ok, but when I run it on hive sql-shell, I
> get a error like below:
>
>
>
> So what can I do for it, specify the partition in ddl ?
>
> liubo07199
>
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=liubo07199&uid=liubo07199%40hellobike.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22liubo07199%40hellobike.com%22%5D>
>
>
Re:(Hive) insert overwrite table xxx partition (pt='xxx') on
iceberg table
Posted by 1 <li...@126.com>.
Error pic cannot upload, hive error is
FAILED: ValidationFailureSemanticException table is not partitioned but partition spec exists: {pt=xxx}
| |
liubo07199
|
|
liubo07199@hellobike.com
|
签名由网易邮箱大师定制
On 05/10/2021 16:07，1<li...@126.com> wrote：
Hi, team
When I migrate tables from hive to iceberg use spark3, partition info in ddl is hidden.
When I run insert overwrite table xxx partition (pt='xxx’) on flink or spark sql-shell, it’s ok, but when I run it on hive sql-shell, I get a error like below:
So what can I do for it, specify the partition in ddl ?
| |
liubo07199
|
|
|