You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Qifan Chen (Jira)" <ji...@apache.org> on 2021/03/30 18:46:00 UTC
[jira] [Comment Edited] (IMPALA-10532)
TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky
[ https://issues.apache.org/jira/browse/IMPALA-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311652#comment-17311652 ]
Qifan Chen edited comment on IMPALA-10532 at 3/30/21, 6:45 PM:
---------------------------------------------------------------
Ran the following script to create a table lineitem_orderkey_only_ec on EC directory
{code:java}
hdfs dfs -rmr -skipTrash /ec_tmp_rs_3_2
hdfs dfs -mkdir /ec_tmp_rs_3_2
~/Impala/bin/impala-shell.sh --protocol=beeswax -f ddl.lineitem_orderkey_only_ec
hdfs ec -setPolicy -path /ec_tmp_rs_3_2 -policy XOR-2-1-1024k
hdfs ec -getPolicy -path /ec_tmp_rs_3_2
~/Impala/bin/impala-shell.sh --protocol=beeswax -f load.lineitem_orderkey_only_ec
hdfs ec -getPolicy -path /ec_tmp_rs_3_2
{code}
Failed to reproduce with the following dml, as the total counter NumStatsFilteredPages is at 281.
{code:java}
set minmax_filter_threshold=0.5;
set MINMAX_FILTERING_LEVEL=PAGE;
set ALLOW_ERASURE_CODED_FILES=true;
set num_nodes=2;
select straight_join a.l_orderkey from
lineitem_orderkey_only_ec a join [SHUFFLE] tpch_parquet.orders b
where a.l_orderkey = b.o_orderkey
and b.o_custkey = 5 order by l_orderkey;
profile;
Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build df3cf5eb8b82e2d044af7009cf5f1e662b4c57ea)
MINMAX_FILTER_THRESHOLD set to 0.5
MINMAX_FILTERING_LEVEL set to PAGE
ALLOW_ERASURE_CODED_FILES set to true
NUM_NODES set to 2
EXPLAIN_LEVEL set to 3
Query: --explain
select straight_join a.l_orderkey from
lineitem_orderkey_only_ec a join [SHUFFLE] tpch_parquet.orders b
where a.l_orderkey = b.o_orderkey
and b.o_custkey = 5 order by l_orderkey
Query submitted at: 2021-03-30 14:43:28 (Coordinator: http://qifan-10229:25000)
Query progress can be monitored at: http://qifan-10229:25000/query_plan?query_id=cd4bb282d9e9df5d:4bda62d700000000
Fetched 19 row(s) in 0.55s
- NumRuntimeFilteredPages: 93 (93)
- NumStatsFilteredPages: 93 (93)
- NumRuntimeFilteredPages: 0 (0)
- NumStatsFilteredPages: 0 (0)
- NumRuntimeFilteredPages: 130 (130)
- NumStatsFilteredPages: 130 (130)
- NumRuntimeFilteredPages: 151 (151)
- NumStatsFilteredPages: 151 (151)
- NumRuntimeFilteredPages: 0 (0)
- NumStatsFilteredPages: 128 (128)
- NumRuntimeFilteredPages: 0 (0)
- NumStatsFilteredPages: 200 (200)
- NumRuntimeFilteredPages: 0 (0)
- NumStatsFilteredPages: 56 (56)
{code}
was (Author: sql_forever):
Ran the following script to create a table lineitem_orderkey_only_ec on EC directory
{code:java}
hdfs dfs -rmr -skipTrash /ec_tmp_rs_3_2
hdfs dfs -mkdir /ec_tmp_rs_3_2
~/Impala/bin/impala-shell.sh --protocol=beeswax -f ddl.lineitem_orderkey_only_ec
hdfs ec -setPolicy -path /ec_tmp_rs_3_2 -policy XOR-2-1-1024k
hdfs ec -getPolicy -path /ec_tmp_rs_3_2
~/Impala/bin/impala-shell.sh --protocol=beeswax -f load.lineitem_orderkey_only_ec
hdfs ec -getPolicy -path /ec_tmp_rs_3_2
{code}
Failed to reproduce with the following dml, as the total counter NumStatsFilteredPages is at 281.
{code:java}
set minmax_filter_threshold=0.5;
set MINMAX_FILTERING_LEVEL=PAGE;
set ALLOW_ERASURE_CODED_FILES=true;
set num_nodes=2;
select straight_join a.l_orderkey from
lineitem_orderkey_only_ec a join [SHUFFLE] tpch_parquet.orders b
where a.l_orderkey = b.o_orderkey
and b.o_custkey = 5 order by l_orderkey;
profile;
{code}
> TestOverlapMinMaxFilters.test_overlap_min_max_filters seems flaky
> ------------------------------------------------------------------
>
> Key: IMPALA-10532
> URL: https://issues.apache.org/jira/browse/IMPALA-10532
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Frontend
> Affects Versions: Impala 4.0
> Reporter: Wenzhe Zhou
> Assignee: Qifan Chen
> Priority: Critical
> Labels: broken-build
> Attachments: ErrorMsg.txt, ErrorMsg2.txt
>
>
> [TestOverlapMinMaxFilters.test_overlap_min_max_filters failed in staging exhaustive release|#showFailuresLink] with following two set of parameters.
>
> query_test.test_runtime_filters.TestOverlapMinMaxFilters.test_overlap_min_max_filters[protocol: beeswax | exec_option: \\{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test.test_runtime_filters.TestOverlapMinMaxFilters.test_overlap_min_max_filters[protocol: beeswax | exec_option: \\{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org