You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Ayush Saxena <ay...@gmail.com> on 2023/10/24 03:03:07 UTC

Re: Hive's performance for querying the Iceberg table is very poor.

Hi Lisoda,

The iceberg jar for hive 3.1.3 doesn't have a lot of changes, We did a
bunch of improvements on the 4.x line for Hive-Iceberg. You can give
iceberg a try on the 4.0.0-beta-1 release mentioned here [1], we have a
bunch of improvements like vecotrization and stuff like that. If you wanna
give it a quick try on docker, we have docker image published for that here
[2] & Iceberg works out of the box there.

Rest feel free to create tickets, if you find some specific queries or
scenarios which are problematic, we will be happy to chase them & get them
sorted.

PS. Not sure about StarRocks, FWIW. That is something we don't develop as
part of Apache Hive nor as part of Apache Software Foundation to best of my
knowledge, so would refrain from or commenting about that on "Apache Hive"
ML

-Ayush


[1] https://hive.apache.org/general/downloads/
[2] https://hub.docker.com/r/apache/hive/tags

On Tue, 24 Oct 2023 at 05:28, Albert Wong <al...@celerdata.com> wrote:

> Too bad.   Tencent Games used StarRocks with Apache Iceberg to power their
> analytics.
> https://medium.com/starrocks-engineering/tencent-games-inside-scoop-the-road-to-cloud-native-with-starrocks-d7dcb2438e25.
>
>
> On Mon, Oct 23, 2023 at 10:55 AM lisoda <li...@yeah.net> wrote:
>
>> We are not going to use starrocks.
>> mpp architecture databases have natural limitations, and starrocks does
>> not necessarily perform better than hive llap.
>>
>>
>> ---- Replied Message ----
>> From Albert Wong<al...@celerdata.com> <al...@celerdata.com>
>> Date 10/24/2023 01:39
>> To user@hive.apache.org
>> Cc
>> Subject Re: Hive's performance for querying the Iceberg table is very
>> poor.
>> I would try http://starrocks.io.   StarRocks is an MPP OLAP database
>> that can query Apache Iceberg and we can cache the data for faster
>> performance.  We also have additional features like building materialized
>> views that span across Apache Iceberg, Apache Hudi and Apache Hive.   Here
>> is a video of connecting the 2 products through a webinar StarRocks did
>> with Tabular (authors of Apache Iceberg).
>> https://www.youtube.com/watch?v=bAmcTrX7hCI&t=10s
>>
>> On Mon, Oct 23, 2023 at 7:18 AM lisoda <li...@yeah.net> wrote:
>>
>>> Hi Team.
>>>       I recently was testing Hive query Iceberg table , I found that
>>> Hive query Iceberg table performance is very very poor . Almost impossible
>>> to use in the production environment . And Join conditions can not be
>>> pushed down to the Iceberg partition.
>>>       I'm using the 1.3.1 Hive Runtime Jar from the Iceberg community.
>>>       Currently I'm using Hive 3.1.3, Iceberg 1.3.1.
>>>       Now I'm very frustrated because the performance is so bad that I
>>> can't deliver to my customers. How can I solve this problem?
>>>      Details:
>>> https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1695050248606629
>>>     I would be grateful if someone could guide me.
>>>
>>

Re:Re: Re: Hive's performance for querying the Iceberg table is very poor.

Posted by lisoda <li...@yeah.net>.

HI.
I tried count(*), and found that the query still doesn't push down to metadata.
Incidentally, I'm using location_based_table. I'm using hadoop_catalog.

在 2023-11-09 10:57:35，"Butao Zhang" <bu...@163.com> 写道：

Hi lisoda. You can check this ticket https://issues.apache.org/jira/browse/HIVE-27347 which can use iceberg basic stats to optimize count(*) query. Note: it didn't take effect if having delete files.

Thanks,

Butao Zhang

---- Replied Message ----
| From | lisoda<li...@yeah.net> |
| Date | 11/9/2023 10:43 |
| To | <us...@hive.apache.org> |
| Subject | Re:Re: Re: Hive's performance for querying the Iceberg table is very poor. |
HI.
I am testing with HIVE-4.0.0-BETA-1 version and I am using location_based_table.
So far I found that HIVE still can't push some queries down to METADATA, e.g. COUNT(*).
Is HIVE 4.0.0-BETA-1 still not able to support query push down?

在 2023-10-24 17:41:20，"Ayush Saxena" <ay...@gmail.com> 写道：

HIVE-27734 is in progress, as I see we have a POC attached to the ticket, we should have it in 2-3 week I believe.

> Also, after the release of 4.0.0, will we be able to do all TPCDS queries on ICEBERG except for normal HIVE tables?

Yep, I believe most of the TPCDS queries would be supported even today on Hive master, but 4.0.0 would have them running for sure.

-Ayush

On Tue, 24 Oct 2023 at 14:51, lisoda <li...@yeah.net> wrote:

Thanks.
I would like to know if hive currently supports push to ICEBERG table partition under JOIN condition.
Because I see HIVE-27734 is not yet complete, what is its progress so far?
Also, after the release of 4.0.0, will we be able to do all TPCDS queries on ICEBERG except for normal HIVE tables?

在 2023-10-24 11:03:07，"Ayush Saxena" <ay...@gmail.com> 写道：

Hi Lisoda,

The iceberg jar for hive 3.1.3 doesn't have a lot of changes, We did a bunch of improvements on the 4.x line for Hive-Iceberg. You can give iceberg a try on the 4.0.0-beta-1 release mentioned here [1], we have a bunch of improvements like vecotrization and stuff like that. If you wanna give it a quick try on docker, we have docker image published for that here [2] & Iceberg works out of the box there.

Rest feel free to create tickets, if you find some specific queries or scenarios which are problematic, we will be happy to chase them & get them sorted.

PS. Not sure about StarRocks, FWIW. That is something we don't develop as part of Apache Hive nor as part of Apache Software Foundation to best of my knowledge, so would refrain from or commenting about that on "Apache Hive" ML

-Ayush

[1] https://hive.apache.org/general/downloads/
[2] https://hub.docker.com/r/apache/hive/tags

On Tue, 24 Oct 2023 at 05:28, Albert Wong <al...@celerdata.com> wrote:

Too bad. Tencent Games used StarRocks with Apache Iceberg to power their analytics. https://medium.com/starrocks-engineering/tencent-games-inside-scoop-the-road-to-cloud-native-with-starrocks-d7dcb2438e25.

On Mon, Oct 23, 2023 at 10:55 AM lisoda <li...@yeah.net> wrote:

We are not going to use starrocks.
mpp architecture databases have natural limitations, and starrocks does not necessarily perform better than hive llap.

---- Replied Message ----
| From | Albert Wong<al...@celerdata.com> |
| Date | 10/24/2023 01:39 |
| To | user@hive.apache.org |
| Cc | |
| Subject | Re: Hive's performance for querying the Iceberg table is very poor. |
I would try http://starrocks.io. StarRocks is an MPP OLAP database that can query Apache Iceberg and we can cache the data for faster performance. We also have additional features like building materialized views that span across Apache Iceberg, Apache Hudi and Apache Hive. Here is a video of connecting the 2 products through a webinar StarRocks did with Tabular (authors of Apache Iceberg). https://www.youtube.com/watch?v=bAmcTrX7hCI&t=10s

On Mon, Oct 23, 2023 at 7:18 AM lisoda <li...@yeah.net> wrote:

Hi Team.
I recently was testing Hive query Iceberg table , I found that Hive query Iceberg table performance is very very poor . Almost impossible to use in the production environment . And Join conditions can not be pushed down to the Iceberg partition.
I'm using the 1.3.1 Hive Runtime Jar from the Iceberg community.
Currently I'm using Hive 3.1.3, Iceberg 1.3.1.
Now I'm very frustrated because the performance is so bad that I can't deliver to my customers. How can I solve this problem?
Details: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1695050248606629
I would be grateful if someone could guide me.

Re: Re: Hive's performance for querying the Iceberg table is very poor.

Posted by Simhadri G <si...@gmail.com>.

Please ensure hive.stats.autogather  is enabled as well.

On Fri, Nov 10, 2023, 2:57 PM Denys Kuzmenko <dk...@apache.org> wrote:

> `hive.iceberg.stats.source` controls where the stats should be sourced
> from. When it's set to iceberg (default), we should go directly to iceberg
> and bypass HMS.
>

Re: Re: Hive's performance for querying the Iceberg table is very poor.

Posted by Denys Kuzmenko <dk...@apache.org>.

`hive.iceberg.stats.source` controls where the stats should be sourced from. When it's set to iceberg (default), we should go directly to iceberg and bypass HMS.

Re: Re: Re: Re: Hive's performance for querying the Iceberg table is very poor.

Posted by Okumin <ma...@okumin.com>.

Hi,

> For iceberg scan performance, Can you share more detail info？I want to
see what's diff between us.

I just quickly & naively created a table of TPC-DS with scale factor = 10GB
on my machine and full-scanned them. Please note that I am not likely to
investigate more details timely because of my knowledge of Iceberg and my
capacity.

```
CREATE TABLE web_sales_orc_zstd STORED AS ORC tblproperties
("orc.compress"="ZSTD") AS SELECT * FROM web_sales;
CREATE TABLE web_sales_orc_snappy STORED AS ORC tblproperties
("orc.compress"="SNAPPY") AS SELECT * FROM web_sales;
CREATE TABLE web_sales_iceberg_zstd STORED BY ICEBERG STORED AS ORC
tblproperties ("orc.compress"="ZSTD") AS SELECT * FROM web_sales;
CREATE TABLE web_sales_iceberg_snappy STORED BY ICEBERG STORED AS ORC
tblproperties ("orc.compress"="SNAPPY") AS SELECT * FROM web_sales;

SELECT * FROM web_sales_orc_zstd WHERE RAND() = 0.0; -- 9.37s
SELECT * FROM web_sales_orc_snappy WHERE RAND() = 0.0; -- 8.63s
SELECT * FROM web_sales_iceberg_zstd WHERE RAND() = 0.0; -- 12.92s
SELECT * FROM web_sales_iceberg_snappy WHERE RAND() = 0.0; -- 11.36s
```

> Thanks Okumin for the info. I think
https://issues.apache.org/jira/browse/HIVE-21520 is the right to fix Query
Execution Summary. Let's try to get it in!

Thank you, Butao!

Regards,
Okumin

On Wed, Dec 20, 2023 at 10:57 AM lisoda <li...@yeah.net> wrote:

> Hi.Okumin
> For iceberg scan performance, Can you share more detail info？I want to see
> what's diff between us.
> Tks.
>
>
>
>
>
> 在 2023-12-16 00:47:05，"Okumin" <ma...@okumin.com> 写道：
>
> Hi,
>
> I don't have any ideas on why Iceberg is slower. I quickly tested ORC +
> ZSTD vs Iceberg + ORC + ZSTD and didn't see significant differences.
>
> As for the issue of Query Execution Summary, I can observe the same issue.
> Maybe, we should merge https://issues.apache.org/jira/browse/HIVE-21520
> to recover the metrics.
>
> Regards,
> Okumin
>
> On Mon, Nov 20, 2023 at 11:52 PM lisoda <li...@yeah.net> wrote:
>
>> Also, I have another question.
>> The Submit Plan statistic is clearly incorrect. Is this something that
>> needs to be fixed?
>>
>>
>>

Re:Re: Re: Re: Hive's performance for querying the Iceberg table is very poor.

Posted by lisoda <li...@yeah.net>.

Hi.Okumin
For iceberg scan performance, Can you share more detail info？I want to see what's diff between us. 
Tks.











在 2023-12-16 00:47:05，"Okumin" <ma...@okumin.com> 写道：

Hi,


I don't have any ideas on why Iceberg is slower. I quickly tested ORC + ZSTD vs Iceberg + ORC + ZSTD and didn't see significant differences.


As for the issue of Query Execution Summary, I can observe the same issue. Maybe, we should merge https://issues.apache.org/jira/browse/HIVE-21520 to recover the metrics.


Regards,
Okumin


On Mon, Nov 20, 2023 at 11:52 PM lisoda <li...@yeah.net> wrote:

Also, I have another question.
The Submit Plan statistic is clearly incorrect. Is this something that needs to be fixed?