You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by bo...@apache.org on 2022/02/24 10:33:43 UTC
[impala] branch master updated (7cfc73c -> 873fe2e)
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 7cfc73c IMPALA-11150: Remove resource-requirements tests on functional_parquet.alltypes
new 8217c23 IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
new 5d516c2 IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
new 873fe2e IMPALA-11135: Deflake LEFT ANTI JOIN test case in test_spilling.py
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/parquet/hdfs-parquet-scanner.cc | 9 +++++++++
.../org/apache/impala/planner/HBaseScanNode.java | 10 ++++++----
testdata/data/README | 5 +++++
testdata/data/partition_col_in_parquet.parquet | Bin 0 -> 691 bytes
.../queries/QueryTest/spilling.test | 4 +++-
tests/query_test/test_runtime_filters.py | 21 +++++++++++++++++++++
6 files changed, 44 insertions(+), 5 deletions(-)
create mode 100644 testdata/data/partition_col_in_parquet.parquet
[impala] 01/03: IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 8217c230acf5a34278794937b69eff97f2755301
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Feb 22 19:27:58 2022 +0100
IMPALA-11147: Min/max filtering crashes on Parquet file that contains partition columns
Impala crashes on a Parquet file that contains the partition columns.
Data files usually don't contain the partition columns, so Impala don't
expect to find such columns in the data files. Unfortunately min/max
filtering generates a SEGFAULT when the partition column is present in
the data files.
It happens when FindSkipRangesForPagesWithMinMaxFilters() tries to
retrieve the Parquet schema element for a given slot descriptor. When
the slot descriptor refers to a partition column, we usually don't find
a schema element so we don't try to skip pages.
But when the partition column is present in the data file, the code
tries to calculate the filtered pages for the column. It uses the column
reader object corresponding to the column, but this is NULL for
partition columns, hence we get a SEGFAULT.
The code shouldn't do anything at the page-level for partition columns,
as the data in such columns are the same for the whole file and it is
already filtered at a higher level.
Testing:
* added e2e test
Change-Id: I17eff4467da3fd67a21353ba2d52d3bec405acd2
Reviewed-on: http://gerrit.cloudera.org:8080/18265
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/exec/parquet/hdfs-parquet-scanner.cc | 9 +++++++++
testdata/data/README | 5 +++++
testdata/data/partition_col_in_parquet.parquet | Bin 0 -> 691 bytes
tests/query_test/test_runtime_filters.py | 21 +++++++++++++++++++++
4 files changed, 35 insertions(+)
diff --git a/be/src/exec/parquet/hdfs-parquet-scanner.cc b/be/src/exec/parquet/hdfs-parquet-scanner.cc
index 94f2b6e..3306b64 100644
--- a/be/src/exec/parquet/hdfs-parquet-scanner.cc
+++ b/be/src/exec/parquet/hdfs-parquet-scanner.cc
@@ -1464,6 +1464,15 @@ Status HdfsParquetScanner::FindSkipRangesForPagesWithMinMaxFilters(
}
int col_idx = node->col_idx;
+
+ if (UNLIKELY(scalar_reader_map_.find(col_idx) == scalar_reader_map_.end())) {
+ // IMPALA-11147: It's possible that 'node' is also a partitioning column. In that
+ // case we don't have a scalar reader for it.
+ DCHECK(scan_node_->hdfs_table()->IsIcebergTable() ||
+ slot_desc->col_pos() < scan_node_->num_partition_keys());
+ continue;
+ }
+
ColumnStatsReader stats_reader =
CreateStatsReader(file_metadata_, row_group, node, slot_desc->type());
diff --git a/testdata/data/README b/testdata/data/README
index f687034..6b928c0 100644
--- a/testdata/data/README
+++ b/testdata/data/README
@@ -678,3 +678,8 @@ field ids.
too_many_def_levels.parquet:
Written by Impala 2.5. The Parquet pages have more encoded def levels than num_values.
+
+partition_col_in_parquet.parquet:
+Written by Impala 4.0. Parquet file with INT and DATE column. Values in the DATE columns
+are identical. There's only a single value per page in the Parquet file (written by
+setting query option 'parquet_page_row_count_limit' to 1).
diff --git a/testdata/data/partition_col_in_parquet.parquet b/testdata/data/partition_col_in_parquet.parquet
new file mode 100644
index 0000000..7ea2625
Binary files /dev/null and b/testdata/data/partition_col_in_parquet.parquet differ
diff --git a/tests/query_test/test_runtime_filters.py b/tests/query_test/test_runtime_filters.py
index bb2649d..d22e522 100644
--- a/tests/query_test/test_runtime_filters.py
+++ b/tests/query_test/test_runtime_filters.py
@@ -17,6 +17,7 @@
#
from copy import deepcopy
+import os
import pytest
import re
import time
@@ -320,6 +321,26 @@ class TestOverlapMinMaxFilters(ImpalaTestSuite):
unique_database,
test_file_vars={'$RUNTIME_FILTER_WAIT_TIME_MS': str(WAIT_TIME_MS)})
+ @SkipIfLocal.hdfs_client
+ def test_partition_column_in_parquet_data_file(self, vector, unique_database):
+ """IMPALA-11147: Test that runtime min/max filters still work on data files that
+ contain the partitioning columns."""
+ tbl_name = "part_col_in_data_file"
+ self.execute_query("CREATE TABLE {0}.{1} (i INT) PARTITIONED BY (d DATE) "
+ "STORED AS PARQUET".format(unique_database, tbl_name))
+ tbl_loc = self._get_table_location("{0}.{1}".format(unique_database, tbl_name),
+ vector)
+ self.filesystem_client.make_dir(tbl_loc[tbl_loc.find('test-warehouse'):] +
+ '/d=2022-02-22/')
+ self.filesystem_client.copy_from_local(os.environ['IMPALA_HOME'] +
+ '/testdata/data/partition_col_in_parquet.parquet', tbl_loc + '/d=2022-02-22/')
+ self.execute_query("ALTER TABLE {0}.{1} RECOVER PARTITIONS".format(
+ unique_database, tbl_name))
+ self.execute_query("SET PARQUET_FALLBACK_SCHEMA_RESOLUTION=NAME")
+ self.execute_query("SET ENABLED_RUNTIME_FILTER_TYPES=MIN_MAX")
+ self.execute_query("select * from {0}.{1} t1, {0}.{1} t2 where t1.d=t2.d and t2.i=2".
+ format(unique_database, tbl_name))
+
# Apply both Bloom filter and Minmax filters
class TestAllRuntimeFilters(ImpalaTestSuite):
@classmethod
[impala] 02/03: IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 5d516c2e0d05c059873a0ce361cd2dcf82ec3477
Author: Qifan Chen <qc...@cloudera.com>
AuthorDate: Thu Feb 17 17:23:35 2022 -0500
IMPALA-11132 Front-end test PlannerTest.testResourceRequirements can fail
This patch addresses the potential row count over-estimation against
HBase tables by capping the estimation by the row count when available
from HMS.
Testing:
1. ran core test successfully
Change-Id: I11c51f76212e1337a7e726097931890c2edab182
Reviewed-on: http://gerrit.cloudera.org:8080/18250
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java b/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
index 17f570f..12c14a3 100644
--- a/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java
@@ -319,11 +319,12 @@ public class HBaseScanNode extends ScanNode {
// May return -1 for the estimate if insufficient data is available.
estimate = tbl.getEstimatedRowStats(startKey_, stopKey_);
}
+ long rowsFromHms = tbl.getTTableStats().getNum_rows();
if (estimate.first == -1) {
// No useful estimate. Rely on HMS row count stats.
// This works only if HBase stats are available in HMS. This is true
// for the Impala tests, and may be true for some applications.
- cardinality_ = tbl.getTTableStats().getNum_rows();
+ cardinality_ = rowsFromHms;
if (LOG.isTraceEnabled()) {
LOG.trace("Fallback to use table stats in HMS: num_rows=" + cardinality_);
}
@@ -336,9 +337,10 @@ public class HBaseScanNode extends ScanNode {
}
} else {
// Use the HBase sampling scan to estimate cardinality. Note that,
- // in tests, this estimate has proven to be very rough: off by
- // 2x or more.
- cardinality_ = estimate.first;
+ // in tests, this estimate has proven to be very rough: off by 2x or more.
+ // Cap the cardinality estimation by the row count from HMS when available.
+ cardinality_ = (rowsFromHms >= 0) ? Long.min(estimate.first, rowsFromHms) :
+ estimate.first;
if (estimate.second > 0) {
suggestedCaching_ = (int)
Math.max(MAX_HBASE_FETCH_BATCH_SIZE / estimate.second, 1);
[impala] 03/03: IMPALA-11135: Deflake LEFT ANTI JOIN test case in test_spilling.py
Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
boroknagyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 873fe2e5241c5714dfd94a186d524edc1cbad0ad
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Mon Feb 21 11:19:39 2022 -0800
IMPALA-11135: Deflake LEFT ANTI JOIN test case in test_spilling.py
TestSpillingDebugActionDimensions.test_spilling has been flaky because a
test case from IMPALA-9725 sometimes does not spill its hash join
partition. This patch lowers the buffer_pool_limit of this test from
110MB to 105MB, just slightly above its Max Per-Host Resource
Reservation (104.61MB), to ensure consistent spilling behavior.
Testing:
After lowering the buffer pool limit, I loop the test 1000 times, and
all spill consistently in fragment "HASH_JOIN_NODE (id=14)".
To be specific, these are the num of SpilledPartitions of the first
instance (ending with "000d") of "Hash Join Builder (join_node_id=14)"
fragment across 1000 query runs:
+--------------------+----------+
| #SpilledPartitions | #Queries |
+--------------------+----------+
| 2 | 30 |
| 3 | 96 |
| 4 | 674 |
| 5 | 52 |
| 6 | 146 |
| 7 | 2 |
+--------------------+----------+
Change-Id: Idad9fc6ec6a0ba7fc70e0701e567da7165e40e83
Reviewed-on: http://gerrit.cloudera.org:8080/18261
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/workloads/functional-query/queries/QueryTest/spilling.test | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/spilling.test b/testdata/workloads/functional-query/queries/QueryTest/spilling.test
index 2632953..5867949 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/spilling.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/spilling.test
@@ -313,7 +313,9 @@ Repartitioning did not reduce the size of a spilled partition
# This test is crafted to always return 0 rows, but with the previous bug
# would sometimes return rows while spilling. The query needs to have a join
# key that is more than 256 bytes wide to trigger the bug.
-set buffer_pool_limit=110m;
+# Need to lower buffer_pool_limit just slightly above the Max Per-Host Resource
+# Reservation (104.61MB) for the join to spill consistently.
+set buffer_pool_limit=105m;
set runtime_filter_mode=0;
SELECT straight_join o_orderkey
FROM (