You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jesus Camacho Rodriguez (JIRA)" <ji...@apache.org> on 2019/07/11 22:47:00 UTC
[jira] [Updated] (HIVE-21509) LLAP may cache corrupted column
vectors and return wrong query result
[ https://issues.apache.org/jira/browse/HIVE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jesus Camacho Rodriguez updated HIVE-21509:
-------------------------------------------
Fix Version/s: 4.0.0
> LLAP may cache corrupted column vectors and return wrong query result
> ---------------------------------------------------------------------
>
> Key: HIVE-21509
> URL: https://issues.apache.org/jira/browse/HIVE-21509
> Project: Hive
> Issue Type: Bug
> Components: llap
> Reporter: Adam Szita
> Assignee: Adam Szita
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21509.0.wip.patch, HIVE-21509.1.wip.patch, HIVE-21509.2.patch, HIVE-21509.3.patch, HIVE-21509.4.patch
>
>
> In some scenarios, LLAP might store column vectors in cache that are getting reused and reset just before their original content would be written.
> The issue is a concurrency issue and is thereby flaky. It is not easy to reproduce, but the odds of surfacing this issue can by improved by setting LLAP executor and IO thread counts this way:
> * set hive.llap.daemon.num.executors=32;
> * set hive.llap.io.threadpool.size=1;
> * using TPCDS input data of store_sales table, have at least a couple of 100k's of rows, and use text format:
> {code:java}
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'{code}
> * having more splits increases the issue showing itself, so it is worth to _set tez.grouping.min-size=1024; set tez.grouping.max-size=1024;_
> * run query on this this table: select min(ss_sold_date_sk) from store_sales;
> The first query result is correct (2450816 in my case). Repeating the query will trigger reading from LLAP cache and produce a wrong result: 0.
> If one wants to make sure of running into this issue, place a Thread.sleep(250) at the beginning of VectorDeserializeOrcWriter#run().
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)