You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Simhadri Govindappa (Jira)" <ji...@apache.org> on 2023/05/08 14:56:00 UTC
[jira] [Created] (HIVE-27327) Iceberg basic stats: Incorrect row count in snapshot summary leading to unoptimized plans
Simhadri Govindappa created HIVE-27327:
------------------------------------------
Summary: Iceberg basic stats: Incorrect row count in snapshot summary leading to unoptimized plans
Key: HIVE-27327
URL: https://issues.apache.org/jira/browse/HIVE-27327
Project: Hive
Issue Type: Bug
Reporter: Simhadri Govindappa
Assignee: Simhadri Govindappa
In the absence of equality deletes, the total row count should be :
{noformat}
row_count = total-records - total-position-deletes{noformat}
Example:
After many inserts and deletes, there are only 46 records in a table.
{noformat}
>>select count(*) from llap_orders;
+------+
| _c0 |
+------+
| 46 |
+------+
1 row selected (7.22 seconds)
{noformat}
But the total records in snapshot summary indicate that there are 300 records
{noformat}
{
"sequence-number" : 19,
"snapshot-id" : 4237525869561629328,
"parent-snapshot-id" : 2572487769557272977,
"timestamp-ms" : 1683553017982,
"summary" : {
"operation" : "append",
"added-data-files" : "5",
"added-records" : "12",
"added-files-size" : "3613",
"changed-partition-count" : "5",
"total-records" : "300",
"total-files-size" : "164405",
"total-data-files" : "100",
"total-delete-files" : "73",
"total-position-deletes" : "254",
"total-equality-deletes" : "0"
}{noformat}
As a result of this , the hive plans generated are unoptimized.
{noformat}
0: jdbc:hive2://simhadrigovindappa-2.simhadri> explain update llap_orders set itemid=7 where itemid=5;
INFO : OK
+----------------------------------------------------+
| Explain |
+----------------------------------------------------+
| Vertex dependency in root stage |
| Reducer 2 <- Map 1 (SIMPLE_EDGE) |
| Reducer 3 <- Map 1 (SIMPLE_EDGE) |
| |
| Stage-4 |
| Stats Work{} |
| Stage-0 |
| Move Operator |
| table:{"name:":"db.llap_orders"} |
| Stage-3 |
| Dependency Collection{} |
| Stage-2 |
| Reducer 2 vectorized |
| File Output Operator [FS_14] |
| table:{"name:":"db.llap_orders"} |
| Select Operator [SEL_13] (rows=150 width=424) |
| Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9"] |
| <-Map 1 [SIMPLE_EDGE] |
| SHUFFLE [RS_4] |
| Select Operator [SEL_3] (rows=150 width=424) |
| Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col7","_col8","_col9"] |
| Select Operator [SEL_2] (rows=150 width=644) |
| Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col13","_col14","_col15"] |
| Filter Operator [FIL_9] (rows=150 width=220) |
| predicate:(itemid = 5) |
| TableScan [TS_0] (rows=300 width=220) |
| db@llap_orders,llap_orders,Tbl:COMPLETE,Col:COMPLETE,Output:["orderid","quantity","itemid","tradets","p1","p2"] |
| Reducer 3 vectorized |
| File Output Operator [FS_16] |
| table:{"name:":"db.llap_orders"} |
| Select Operator [SEL_15] |
| Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col4","_col5"] |
| <-Map 1 [SIMPLE_EDGE] |
| SHUFFLE [RS_10] |
| PartitionCols:_col4, _col5 |
| Select Operator [SEL_7] (rows=150 width=220) |
| Output:["_col0","_col1","_col2","_col3","_col4","_col5"] |
| Please refer to the previous Select Operator [SEL_2] |
| |
+----------------------------------------------------+
39 rows selected (0.104 seconds)
0: jdbc:hive2://simhadrigovindappa-2.simhadri>{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)