[jira] [Updated] (HIVE-12989) Hive with ORC - Data read from disk for 2 queries same despite very different selectivities

["Vishesh Garg (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-12989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vishesh Garg updated HIVE-12989:
--------------------------------
    Attachment: HIVE LOGS.zip

Attaching the DEBUG logs for the two queries.

> Hive with ORC - Data read from disk for 2 queries same despite very different selectivities
> -------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12989
>                 URL: https://issues.apache.org/jira/browse/HIVE-12989
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 1.2.1
>         Environment: TPCDS 100 GB
>            Reporter: Vishesh Garg
>            Priority: Minor
>         Attachments: HIVE LOGS.zip
>
>
> I'm using Hive 1.2.1 with ORC file format and doing some profiling for the data read from the disk (after clearing file system caches) for 2 queries, the data being loaded from the TPCDS benchmark. The queries are:
> {code:sql}
>  Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between 2450816 and 2450817;
>  Q2) select  count(ss_item_sk) from store_sales where ss_sold_date_sk between 2450816 and 2452000;
> {code}
> I'm using flags *hive.vectorized.execution.enabled* = true and *hive.optimize.index.filter* = true.
> The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking at the hive logs, I see that about 11000 row-groups are included for Q1 and 33000 row-groups are included for Q2 after filtering using index. Hence, I had expected vastly different sizes of data read. However, I see about 1GB of data read for both the queries. I suspect this has something to do with the function _addRgFilteredStreamToRanges()_



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)