You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Brown (JIRA)" <ji...@apache.org> on 2018/05/04 16:22:00 UTC
[jira] [Comment Edited] (IMPALA-6781) Stress tests should take into
account of non-deterministic results of queries
[ https://issues.apache.org/jira/browse/IMPALA-6781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464086#comment-16464086 ]
Michael Brown edited comment on IMPALA-6781 at 5/4/18 4:21 PM:
---------------------------------------------------------------
After a bunch of delays last and this week, I'm looking into this again. It's pretty clear what needs to happen, but while I"m at it, I'm checking all other TPCH queries the stress test runs on the regular so that this won't happen again. Note that we've never seen this on an 8-node / tpch 100 scale. It took 140-node / tpch 10000.
was (Author: mikesbrown):
After a bunch of delays last and this week, I'm looking into this again. It's pretty clear what needs to happen, but while I"m at it, I'm checking all other TPCH queries the stress test runs on the regular so that this won't happen again. Note that we've never seen this on an 8-node / tpch 100 scale. It took 14-node / tpch 10000.
> Stress tests should take into account of non-deterministic results of queries
> -----------------------------------------------------------------------------
>
> Key: IMPALA-6781
> URL: https://issues.apache.org/jira/browse/IMPALA-6781
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Reporter: Michael Ho
> Assignee: Michael Brown
> Priority: Critical
>
> Certain queries have the pattern of ordery-by with a limit clause. Due to the limit clause, we may not include all the rows for a given order-by keys in the output. It's non-deterministic that which rows will be included in the output. Given the stress test will store the hashes of the query result obtained during binary search, we may sometimes get false alarm on result mis-match. One way to fix it is that we need to order all the columns so that the results will always be deterministic.
> For instance, the following TPC-H Q3 query may have multiple correct results:
> {noformat}
> Default Db: tpch_10000_parquet
> Sql Statement: /* Mem: 8152 MB. Coordinator: vc1503.halxg.cloudera.com. */
> select
> l_orderkey,
> sum(l_extendedprice * (1 - l_discount)) as revenue,
> o_orderdate,
> o_shippriority
> from
> customer,
> orders,
> lineitem
> where
> c_mktsegment = 'BUILDING'
> and c_custkey = o_custkey
> and l_orderkey = o_orderkey
> and o_orderdate < '1995-03-15'
> and l_shipdate > '1995-03-15'
> group by
> l_orderkey,
> o_orderdate,
> o_shippriority
> order by
> revenue desc,
> o_orderdate
> limit 10
> {noformat}
> Correct result 1:
> {noformat}
> +-------------+-------------+-------------+----------------+
> | l_orderkey | revenue | o_orderdate | o_shippriority |
> +-------------+-------------+-------------+----------------+
> | 18134335556 | 510414.0360 | 1995-02-13 | 0 |
> | 52494073892 | 510414.0360 | 1995-02-13 | 0 |
> | 26724270146 | 510414.0360 | 1995-02-13 | 0 |
> | 954466400 | 510414.0360 | 1995-02-13 | 0 |
> | 9544400966 | 510414.0360 | 1995-02-13 | 0 |
> | 35314204736 | 510414.0360 | 1995-02-13 | 0 |
> | 43904139302 | 510414.0360 | 1995-02-13 | 0 |
> | 7133834945 | 503732.3318 | 1995-02-25 | 0 |
> | 24313704101 | 503732.3318 | 1995-02-25 | 0 |
> | 58673442437 | 503732.3318 | 1995-02-25 | 0 |
> +-------------+-------------+-------------+----------------+
> {noformat}
> Correct result 2:
> {noformat}
> +-------------+-------------+-------------+----------------+
> | l_orderkey | revenue | o_orderdate | o_shippriority |
> +-------------+-------------+-------------+----------------+
> | 26724270146 | 510414.0360 | 1995-02-13 | 0 |
> | 43904139302 | 510414.0360 | 1995-02-13 | 0 |
> | 35314204736 | 510414.0360 | 1995-02-13 | 0 |
> | 52494073892 | 510414.0360 | 1995-02-13 | 0 |
> | 18134335556 | 510414.0360 | 1995-02-13 | 0 |
> | 9544400966 | 510414.0360 | 1995-02-13 | 0 |
> | 954466400 | 510414.0360 | 1995-02-13 | 0 |
> | 32903638691 | 503732.3318 | 1995-02-25 | 0 |
> | 15723769511 | 503732.3318 | 1995-02-25 | 0 |
> | 50083507847 | 503732.3318 | 1995-02-25 | 0 |
> +-------------+-------------+-------------+----------------+
> {noformat}
> If one changes the limit clause to 15, we will see that the last 3 rows should be a subset of rows with (503732.3318, 1995-02-25) as the ordering keys {{revenue}} and {{o_orderdate}} are identical so it's non-deterministic which rows are shown. The result will be deterministic if the order-by clause also include {{l_orderkey}} and {{o_shippriority}}
> {noformat}
> +-------------+-------------+-------------+----------------+
> | l_orderkey | revenue | o_orderdate | o_shippriority |
> +-------------+-------------+-------------+----------------+
> | 26724270146 | 510414.0360 | 1995-02-13 | 0 |
> | 43904139302 | 510414.0360 | 1995-02-13 | 0 |
> | 35314204736 | 510414.0360 | 1995-02-13 | 0 |
> | 52494073892 | 510414.0360 | 1995-02-13 | 0 |
> | 18134335556 | 510414.0360 | 1995-02-13 | 0 |
> | 9544400966 | 510414.0360 | 1995-02-13 | 0 |
> | 954466400 | 510414.0360 | 1995-02-13 | 0 |
> | 32903638691 | 503732.3318 | 1995-02-25 | 0 |
> | 15723769511 | 503732.3318 | 1995-02-25 | 0 |
> | 50083507847 | 503732.3318 | 1995-02-25 | 0 |
> | 7133834945 | 503732.3318 | 1995-02-25 | 0 |
> | 41493573281 | 503732.3318 | 1995-02-25 | 0 |
> | 24313704101 | 503732.3318 | 1995-02-25 | 0 |
> | 58673442437 | 503732.3318 | 1995-02-25 | 0 |
> | 54409232481 | 499104.0645 | 1995-03-10 | 0 |
> +-------------+-------------+-------------+----------------+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org