You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jr...@apache.org on 2018/01/11 22:30:52 UTC
[1/3] impala git commit: IMPALA-6330,
IMPALA-5702: Avoid boost's trim() to workaround crash after dynamic
linking.
Repository: impala
Updated Branches:
refs/heads/master c0c1202db -> b27537a15
IMPALA-6330, IMPALA-5702: Avoid boost's trim() to workaround crash after dynamic linking.
Replaces boost::algorithm::trim() with std::string methods when parsing
/proc/self/smaps and adds a trivial unit test for MemInfo::ParseSmaps().
I did *not* replace other uses of trim() with equivalents from
be/src/gutil/strings/strip.h at this moment.
The backstory here is that
TestAdmissionControllerStress::test_admission_controller_with_flags
fails occasionally on dynamically linked builds of Impala. I was able
to reproduce the failure reliably (within 3 tries) with the following:
$ ./buildall.sh -notests -so -noclean
$ bin/start-impala-cluster.py --impalad_args="--memory_maintenance_sleep_time_ms=1"
$ impala-shell.sh --query 'select max(t.c1), avg(t.c2), min(t.c3), avg(c4), avg(c5), avg(c6) from (select max(tinyint_col) over (order by int_col) c1, avg(tinyint_col) over (order by smallint_col) c2, min(tinyint_col) over (order by smallint_col desc) c3, rank() over (order by int_col desc) c4, dense_rank() over (order by bigint_col) c5, first_value(tinyint_col) over (order by bigint_col desc) c6 from functional.alltypes) t;'
The stack trace looks like:
(gdb) bt
#0 0x00007fe230df2428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fe230df402a in __GI_abort () at abort.c:89
#2 0x00007fe23312026d in __gnu_cxx::__verbose_terminate_handler() () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x00007fe2330d8b66 in __cxxabiv1::__terminate(void (*)()) (handler=<optimized out>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00007fe2330d8bb1 in std::terminate() () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x00007fe2330d8cb8 in __cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*)) (obj=0x8e54080, tinfo=0x7fe233356210 <typeinfo for std::bad_cast>, dest=0x7fe23311ea70 <std::bad_cast::~bad_cast()>) at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6 0x00007fe233110332 in std::__throw_bad_cast() () at ../../../../../gcc-4.9.2/libstdc++-v3/src/c++11/functexcept.cc:63
#7 0x00007fe2330e8ad7 in std::use_facet<std::ctype<char> >(std::locale const&) (__loc=...) at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/locale_classes.tcc:137
#8 0x00000000008d2cdf in void boost::algorithm::trim<std::string>(std::string&, std::locale const&) ()
#9 0x00007fe2396d5057 in impala::MemInfo::ParseSmaps() () at /home/philip/src/Impala/be/src/util/mem-info.cc:132
...
My best theory is that there's a race/bug, wherein the std::locale* static initialization
work is getting somehow 'reset' by the dynamic linker, when more libraries are linked
in as a result of the query. My evidence to support this theory is scant, but
I do notice that LD_DEBUG=all prints the following when the query is executed
(but not right at startup):
binding file /home/philip/src/Impala/toolchain/gcc-4.9.2/lib64/libstdc++.so.6 [0] to
/home/philip/src/Impala/toolchain/gflags-2.2.0-p1/lib/libgflags.so.2.2 [0]:
normal symbol `std::locale::facet::_S_destroy_c_locale(__locale_struct*&)'
Note that there are BSS segments for some of std::locale::facet::* inside
of libgflags.so.
$nm toolchain/gflags-2.2.0-p1/lib/libgflags.so | c++filt | grep facet | grep ' B '
00000000002e2d10 B std::locale::facet::_S_c_locale
00000000002e2d0c B std::locale::facet::_S_once
I'm not the first to run into variants of these issues, though the results
are fairly unhelpful:
http://www.boost.org/doc/libs/1_58_0/libs/locale/doc/html/faq.html
https://stackoverflow.com/questions/26990412/c-boost-crashes-while-using-locale
https://svn.boost.org/trac10/ticket/4671
http://clang-developers.42468.n3.nabble.com/std-use-facet-lt-std-ctype-lt-char-gt-gt-crashes-on-linux-td4033967.html
https://unix.stackexchange.com/questions/719/can-we-get-compiler-information-from-an-elf-binary
https://stackoverflow.com/questions/42376100/linking-with-library-causes-collate-facet-to-be-missing-from-char
http://lists.llvm.org/pipermail/cfe-dev/2012-July/023289.html
https://gcc.gnu.org/ml/libstdc++/2014-11/msg00122.html
Change-Id: I8dd807f869a9359d991ba515177fb2298054520e
Reviewed-on: http://gerrit.cloudera.org:8080/8888
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/604e48d2
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/604e48d2
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/604e48d2
Branch: refs/heads/master
Commit: 604e48d2f3f86b18f7d588d1861e17f177dfedc9
Parents: c0c1202
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Tue Dec 19 10:51:57 2017 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Jan 11 08:05:30 2018 +0000
----------------------------------------------------------------------
be/src/util/mem-info.cc | 11 ++++++-----
be/src/util/proc-info-test.cc | 8 ++++++++
2 files changed, 14 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/604e48d2/be/src/util/mem-info.cc
----------------------------------------------------------------------
diff --git a/be/src/util/mem-info.cc b/be/src/util/mem-info.cc
index 4d551f1..9ffc98d 100644
--- a/be/src/util/mem-info.cc
+++ b/be/src/util/mem-info.cc
@@ -36,7 +36,6 @@
using boost::algorithm::is_any_of;
using boost::algorithm::split;
-using boost::algorithm::trim;
using boost::algorithm::token_compress_on;
namespace impala {
@@ -122,14 +121,16 @@ MappedMemInfo MemInfo::ParseSmaps() {
++result.num_maps;
continue;
}
- // Line is in the form of <Name>: <value>, e.g.:
- // Size: 1084 kB
+ // Line is in the form of <Name>:<spaces><value>, e.g.:
+ // Size: 1084 kB
// VmFlags: rd ex mr mw me dw
size_t colon_pos = line.find(':');
if (colon_pos == string::npos) continue;
string name = line.substr(0, colon_pos);
- string value = line.substr(colon_pos + 1, string::npos);
- trim(value);
+ size_t non_space_after_colon_pos = line.find_first_not_of(" ", colon_pos + 1);
+ if (non_space_after_colon_pos == string::npos) continue;
+ // From the first non-space after the colon through the end of the string.
+ string value = line.substr(non_space_after_colon_pos);
// Use atol() to parse the value, ignoring " kB" suffix.
if (name == "Size") {
http://git-wip-us.apache.org/repos/asf/impala/blob/604e48d2/be/src/util/proc-info-test.cc
----------------------------------------------------------------------
diff --git a/be/src/util/proc-info-test.cc b/be/src/util/proc-info-test.cc
index 7946e5b..9e25663 100644
--- a/be/src/util/proc-info-test.cc
+++ b/be/src/util/proc-info-test.cc
@@ -47,6 +47,14 @@ TEST(ProcessStateInfo, Basic) {
ASSERT_GT(process_state_info.GetInt("fd/count"), 0);
}
+TEST(MappedMapInfo, Basic) {
+ MappedMemInfo result = MemInfo::ParseSmaps();
+ ASSERT_GT(result.num_maps, 0);
+ ASSERT_GT(result.size_kb, 0);
+ ASSERT_GT(result.rss_kb, 0);
+ ASSERT_GE(result.anon_huge_pages_kb, 0);
+}
+
}
int main(int argc, char **argv) {
[2/3] impala git commit: Fix typo in test_observability.
Posted by jr...@apache.org.
Fix typo in test_observability.
Should fix "NameError: global name 'dgb_str' is not defined".
Change-Id: Ida3f355c6c6be5ed52e4d445f8f80665cdc8e2b8
Reviewed-on: http://gerrit.cloudera.org:8080/9003
Reviewed-by: Alex Behm <al...@cloudera.com>
Reviewed-by: Zoram Thanga <zo...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/ab81c48d
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/ab81c48d
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/ab81c48d
Branch: refs/heads/master
Commit: ab81c48d7ae6f9466cefb26e023aed020a58d931
Parents: 604e48d
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Wed Jan 10 20:16:35 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Jan 11 20:10:01 2018 +0000
----------------------------------------------------------------------
tests/query_test/test_observability.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/ab81c48d/tests/query_test/test_observability.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_observability.py b/tests/query_test/test_observability.py
index 1396ac5..b25d88e 100644
--- a/tests/query_test/test_observability.py
+++ b/tests/query_test/test_observability.py
@@ -174,4 +174,4 @@ class TestObservability(ImpalaTestSuite):
dbg_str = 'Debug thrift profile for query ' + str(query_id) + ' not available in '
dbg_str += str(MAX_RETRIES) + ' seconds, '
dbg_str += '(' + start_time + ', ' + end_time + ').'
- assert False, dgb_str
+ assert False, dbg_str
[3/3] impala git commit: IMPALA-4252: [DOCS] Document min/max filters
for Kudu tables
Posted by jr...@apache.org.
IMPALA-4252: [DOCS] Document min/max filters for Kudu tables
Change-Id: I15d8c952ab5b90e89fdd57640dfb4da882f7ecb2
Reviewed-on: http://gerrit.cloudera.org:8080/8986
Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/b27537a1
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/b27537a1
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/b27537a1
Branch: refs/heads/master
Commit: b27537a15b679c0c84b7d2e9f49c9a51f9ae93ba
Parents: ab81c48
Author: John Russell <jr...@cloudera.com>
Authored: Tue Jan 9 13:52:02 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Jan 11 21:49:23 2018 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 6 +++++
.../impala_disable_row_runtime_filtering.xml | 14 ++++++++++++
docs/topics/impala_kudu.xml | 17 ++++++++++++++
docs/topics/impala_max_num_runtime_filters.xml | 4 ++++
.../topics/impala_runtime_bloom_filter_size.xml | 4 ++++
docs/topics/impala_runtime_filter_max_size.xml | 4 ++++
docs/topics/impala_runtime_filter_min_size.xml | 4 ++++
docs/topics/impala_runtime_filtering.xml | 24 +++++++++++++++-----
8 files changed, 71 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index dc8cdb5..c1496c6 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1194,6 +1194,12 @@ drop database temp;
<codeph>hadoop fs -cp</codeph>, or <codeph>INSERT</codeph> in Impala or Hive.
</p>
+ <p rev="2.11.0 IMPALA-4252" id="filter_option_bloom_only">
+ This query option affects only Bloom filters, not the min/max filters
+ that are applied to Kudu tables. Therefore, it does not affect the
+ performance of queries against Kudu tables.
+ </p>
+
<p rev="2.9.0 IMPALA-5333" id="adls_dml_performance">
<draft-comment>
Currently nothing to say on this subject. Leaving this placeholder
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_disable_row_runtime_filtering.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_row_runtime_filtering.xml b/docs/topics/impala_disable_row_runtime_filtering.xml
index 3280084..8bdbcda 100644
--- a/docs/topics/impala_disable_row_runtime_filtering.xml
+++ b/docs/topics/impala_disable_row_runtime_filtering.xml
@@ -72,6 +72,20 @@ under the License.
unsetting it immediately afterward.
</p>
+ <p conref="../shared/impala_common.xml#common/file_format_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252">
+ This query option only applies to queries against HDFS-based tables
+ using the Parquet file format.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252">
+ When applied to a query involving a Kudu table, this option turns off
+ all runtime filtering for the Kudu table.
+ </p>
+
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_runtime_filtering.xml"/>,
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_kudu.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_kudu.xml b/docs/topics/impala_kudu.xml
index 08d3559..4260c56 100644
--- a/docs/topics/impala_kudu.xml
+++ b/docs/topics/impala_kudu.xml
@@ -1373,6 +1373,23 @@ kudu.table_name | impala::some_database.table_name_demo
parallelize the query very efficiently.
</p>
+ <p rev="2.11.0 IMPALA-4252">
+ In <keyword keyref="impala211_full"/> and higher, Impala can push down additional
+ information to optimize join queries involving Kudu tables. If the join clause
+ contains predicates of the form
+ <codeph><varname>column</varname> = <varname>expression</varname></codeph>,
+ after Impala constructs a hash table of possible matching values for the
+ join columns from the bigger table (either an HDFS table or a Kudu table), Impala
+ can <q>push down</q> the minimum and maximum matching column values to Kudu,
+ so that Kudu can more efficiently locate matching rows in the second (smaller) table.
+ These min/max filters are affected by the <codeph>RUNTIME_FILTER_MODE</codeph>,
+ <codeph>RUNTIME_FILTER_WAIT_TIME_MS</codeph>, and <codeph>DISABLE_ROW_RUNTIME_FILTERING</codeph>
+ query options; the min/max filters are not affected by the
+ <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph>, <codeph>RUNTIME_FILTER_MIN_SIZE</codeph>,
+ <codeph>RUNTIME_FILTER_MAX_SIZE</codeph>, and <codeph>MAX_NUM_RUNTIME_FILTERS</codeph>
+ query options.
+ </p>
+
<p>
See <xref keyref="explain"/> for examples of evaluating the effectiveness of
the predicate pushdown for a specific query against a Kudu table.
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_max_num_runtime_filters.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_num_runtime_filters.xml b/docs/topics/impala_max_num_runtime_filters.xml
index 7ac06ee..0fa52e0 100644
--- a/docs/topics/impala_max_num_runtime_filters.xml
+++ b/docs/topics/impala_max_num_runtime_filters.xml
@@ -67,6 +67,10 @@ under the License.
<p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+ <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252" conref="../shared/impala_common.xml#common/filter_option_bloom_only"/>
+
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_runtime_filtering.xml"/>,
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_runtime_bloom_filter_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_bloom_filter_size.xml b/docs/topics/impala_runtime_bloom_filter_size.xml
index 62d0c03..96f0ca5 100644
--- a/docs/topics/impala_runtime_bloom_filter_size.xml
+++ b/docs/topics/impala_runtime_bloom_filter_size.xml
@@ -98,6 +98,10 @@ under the License.
unsetting it immediately afterward.
</p>
+ <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252" conref="../shared/impala_common.xml#common/filter_option_bloom_only"/>
+
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_runtime_filtering.xml"/>,
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_runtime_filter_max_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_max_size.xml b/docs/topics/impala_runtime_filter_max_size.xml
index 7e86914..2370d44 100644
--- a/docs/topics/impala_runtime_filter_max_size.xml
+++ b/docs/topics/impala_runtime_filter_max_size.xml
@@ -57,6 +57,10 @@ under the License.
<p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+ <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252" conref="../shared/impala_common.xml#common/filter_option_bloom_only"/>
+
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_runtime_filtering.xml"/>,
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_runtime_filter_min_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_min_size.xml b/docs/topics/impala_runtime_filter_min_size.xml
index c341b79..36a4426 100644
--- a/docs/topics/impala_runtime_filter_min_size.xml
+++ b/docs/topics/impala_runtime_filter_min_size.xml
@@ -57,6 +57,10 @@ under the License.
<p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+ <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+
+ <p rev="2.11.0 IMPALA-4252" conref="../shared/impala_common.xml#common/filter_option_bloom_only"/>
+
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_runtime_filtering.xml"/>,
http://git-wip-us.apache.org/repos/asf/impala/blob/b27537a1/docs/topics/impala_runtime_filtering.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filtering.xml b/docs/topics/impala_runtime_filtering.xml
index 3323afb..ae0f1b4 100644
--- a/docs/topics/impala_runtime_filtering.xml
+++ b/docs/topics/impala_runtime_filtering.xml
@@ -169,16 +169,23 @@ under the License.
of values for join key columns. When this list is values is transmitted in time to a scan node,
Impala can filter out non-matching values immediately after reading them, rather than transmitting
the raw data to another host to compare against the in-memory hash table on that host.
- This data structure is implemented as a <term>Bloom filter</term>, which uses a probability-based
- algorithm to determine all possible matching values. (The probability-based aspects means that the
- filter might include some non-matching values, but if so, that does not cause any inaccuracy
+ </p>
+ <p>
+ For HDFS-based tables, this data structure is implemented as a <term>Bloom filter</term>, which uses
+ a probability-based algorithm to determine all possible matching values. (The probability-based aspects
+ means that the filter might include some non-matching values, but if so, that does not cause any inaccuracy
in the final results.)
</p>
+ <p rev="2.11.0 IMPALA-4252">
+ Another kind of filter is the <q>min-max</q> filter. It currently only applies to Kudu tables. The
+ filter is a data structure representing a minimum and maximum value. These filters are passed to
+ Kudu to reduce the number of rows returned to Impala when scanning the probe side of the join.
+ </p>
<p>
There are different kinds of filters to match the different kinds of joins (partitioned and broadcast).
- A broadcast filter is a complete list of relevant values that can be immediately evaluated by a scan node.
- A partitioned filter is a partial list of relevant values (based on the data processed by one host in the
- cluster); all the partitioned filters must be combined into one (by the coordinator node) before the
+ A broadcast filter reflects the complete list of relevant values and can be immediately evaluated by a scan node.
+ A partitioned filter reflects only the values processed by one host in the
+ cluster; all the partitioned filters must be combined into one (by the coordinator node) before the
scan nodes can use the results to accurately filter the data as it is read from storage.
</p>
<p>
@@ -331,6 +338,9 @@ under the License.
<codeph>runtime filters: <varname>filter_id</varname> <- <varname>table</varname>.<varname>column</varname></codeph>,
while a plan fragment that consumes a filter includes an annotation such as
<codeph>runtime filters: <varname>filter_id</varname> -> <varname>table</varname>.<varname>column</varname></codeph>.
+ <ph rev="2.11.0 IMPALA-4252">Setting the query option <codeph>EXPLAIN_LEVEL=2</codeph> adds additional
+ annotations showing the type of the filter, either <codeph><varname>filter_id</varname>[bloom]</codeph>
+ (for HDFS-based tables) or <codeph><varname>filter_id</varname>[min_max]</codeph> (for Kudu tables).</ph>
</p>
<p>
@@ -507,6 +517,8 @@ select c1 from huge_t1 join [shuffle] huge_t2
The runtime filtering feature is most effective for the Parquet file formats.
For other file formats, filtering only applies for partitioned tables.
See <xref href="impala_runtime_filtering.xml#runtime_filtering_file_formats"/>.
+ For the ways in which runtime filtering works for Kudu tables, see
+ <xref href="impala_kudu.xml#kudu_performance"/>.
</p>
<!-- To do: check if this restriction is lifted in 5.8 / 2.6. -->