You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2022/04/13 15:30:34 UTC
[impala] branch master updated (011bef624 -> 3627b027f)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 011bef624 IMPALA-11241: Mark test_statement_expression_limit as serialized
new 3843f7ff4 IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
new 3627b027f IMPALA-11229: Upgrade Spring framework to 5.3.18
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/aggregator.cc | 3 ++-
be/src/exec/exec-node.cc | 8 +++++---
java/pom.xml | 7 +++----
.../QueryTest/nested-types-subplan-single-node.test | 20 ++++++++++++++++++++
.../QueryTest/union-const-scalar-expr-codegen.test | 6 +++---
tests/common/test_result_verifier.py | 16 ++++++++++++++--
6 files changed, 47 insertions(+), 13 deletions(-)
[impala] 02/02: IMPALA-11229: Upgrade Spring framework to 5.3.18
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 3627b027fea9ba25f204f4166bce90d76e995724
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Wed Apr 6 09:49:34 2022 -0700
IMPALA-11229: Upgrade Spring framework to 5.3.18
This upgrade the Spring framework to 5.3.18 to
address multiple CVEs:
- CVE-2022-22965
- CVE-2022-22950
- CVE-2021-22060
Testing:
- Ran core job
- Ran custom cluster tests in exhaustive mode
Change-Id: Ie1b299c5b24e70c9db6eb0ce37fee9e32908423e
Reviewed-on: http://gerrit.cloudera.org:8080/18405
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
---
java/pom.xml | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/java/pom.xml b/java/pom.xml
index 60109686a..dbb931325 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -69,12 +69,11 @@ under the License.
<iceberg.version>${env.IMPALA_ICEBERG_VERSION}</iceberg.version>
<pac4j.version>4.5.5</pac4j.version>
<!-- xmlsec, bcprov-jdk15on and springframework are not used by Impala directly,
- but they are needed by pac4j. This uses a newer xmlsec to address a CVE,
- but bcprov-jdk15on and springframework versions match the versions from
- pac4j 4.5.5. -->
+ but they are needed by pac4j. This uses a newer xmlsec/spring to address CVEs,
+ but bcprov-jdk15on matches the versions from pac4j 4.5.5. -->
<xmlsec.version>2.2.3</xmlsec.version>
<bcprov-jdk15on.version>1.68</bcprov-jdk15on.version>
- <springframework.version>5.2.9.RELEASE</springframework.version>
+ <springframework.version>5.3.18</springframework.version>
<json-smart.version>2.4.7</json-smart.version>
</properties>
[impala] 01/02: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 3843f7ff46204a5cf1c8c9ed26b9acd9ab28c183
Author: Csaba Ringhofer <cs...@cloudera.com>
AuthorDate: Tue Apr 5 16:55:31 2022 +0200
IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Before this patch the message was added to the profile in Open(),
which can be called multiple times in subplans.
Moved it to Close(), which is only called once in the lifetime
of a Node/Aggregator.
A drawback of this is that this info won't be visible when the
Node is still active, but I don't think that it is a very useful
info in a still running query.
Also added a new feature to test_result_verifier.py:
Inside RUNTIME_PROFILE section row_regex can be negated with !,
so !row_regex [regex] means that regex is not matched by any line
in the profile.
Testing:
- added a regression test
Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f
Reviewed-on: http://gerrit.cloudera.org:8080/18385
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/exec/aggregator.cc | 3 ++-
be/src/exec/exec-node.cc | 8 +++++---
.../QueryTest/nested-types-subplan-single-node.test | 20 ++++++++++++++++++++
.../QueryTest/union-const-scalar-expr-codegen.test | 6 +++---
tests/common/test_result_verifier.py | 16 ++++++++++++++--
5 files changed, 44 insertions(+), 9 deletions(-)
diff --git a/be/src/exec/aggregator.cc b/be/src/exec/aggregator.cc
index d6eb98602..cf3e208f3 100644
--- a/be/src/exec/aggregator.cc
+++ b/be/src/exec/aggregator.cc
@@ -122,7 +122,6 @@ Status Aggregator::Prepare(RuntimeState* state) {
}
Status Aggregator::Open(RuntimeState* state) {
- runtime_profile_->AppendExecOption(config_.codegen_status_msg_);
RETURN_IF_ERROR(AggFnEvaluator::Open(agg_fn_evals_, state));
RETURN_IF_ERROR(ScalarExprEvaluator::Open(conjunct_evals_, state));
return Status::OK();
@@ -137,6 +136,8 @@ void Aggregator::Close(RuntimeState* state) {
if (expr_results_pool_.get() != nullptr) expr_results_pool_->FreeAll();
if (expr_mem_tracker_.get() != nullptr) expr_mem_tracker_->Close();
if (mem_tracker_.get() != nullptr) mem_tracker_->Close();
+
+ runtime_profile_->AppendExecOption(config_.codegen_status_msg_);
}
// TODO: codegen this function.
diff --git a/be/src/exec/exec-node.cc b/be/src/exec/exec-node.cc
index 26ac4236b..1d56e5f27 100644
--- a/be/src/exec/exec-node.cc
+++ b/be/src/exec/exec-node.cc
@@ -290,9 +290,6 @@ Status ExecNode::Prepare(RuntimeState* state) {
Status ExecNode::Open(RuntimeState* state) {
RETURN_IF_ERROR(ExecDebugAction(TExecNodePhase::OPEN, state));
DCHECK_EQ(conjunct_evals_.size(), conjuncts_.size());
- for (const string& codegen_msg : plan_node_.codegen_status_msgs_) {
- runtime_profile_->AppendExecOption(codegen_msg);
- }
return ScalarExprEvaluator::Open(conjunct_evals_, state);
}
@@ -330,6 +327,11 @@ void ExecNode::Close(RuntimeState* state) {
}
mem_tracker_->Close();
}
+
+ for (const string& codegen_msg : plan_node_.codegen_status_msgs_) {
+ runtime_profile_->AppendExecOption(codegen_msg);
+ }
+
if (events_ != nullptr) events_->MarkEvent("Closed");
}
diff --git a/testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan-single-node.test b/testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan-single-node.test
index 37b608f73..135825fa0 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan-single-node.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan-single-node.test
@@ -16,3 +16,23 @@ bigint,bigint
---- RUNTIME_PROFILE
row_regex: .*NUM_NODES=1.*
====
+---- QUERY
+# Regression test for IMPALA-11200
+select c_custkey, v.* from customer c,
+ (select o_orderpriority, count(o_orderkey) c
+ from c.c_orders
+ group by o_orderpriority) v
+where c_custkey < 4;
+---- RESULTS
+2,'2-HIGH',1
+2,'4-NOT SPECIFIED',2
+2,'1-URGENT',4
+1,'2-HIGH',1
+1,'5-LOW',2
+1,'1-URGENT',2
+1,'3-MEDIUM',1
+---- TYPES
+BIGINT,STRING,BIGINT
+---- RUNTIME_PROFILE
+!row_regex: .*Codegen Enabled, Codegen Enabled.*
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test b/testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test
index e8398ebac..732f0becf 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test
@@ -16,7 +16,7 @@ tinyint,tinyint,tinyint
#SORT_NODE
ExecOption: Codegen Enabled
#UNION_NODE
-ExecOption: Codegen Enabled, Codegen Disabled for const scalar expressions
+ExecOption: Codegen Disabled for const scalar expressions, Codegen Enabled
====
---- QUERY
# Test insert statement with values (translated into UNION with const expressions).
@@ -29,7 +29,7 @@ insert into test_values_codegen(c1) values (CAST(1+ceil(2.5)*3 as tinyint));
00:UNION
constant-operands=1
#UNION_NODE
-ExecOption: Codegen Enabled, Codegen Disabled for const scalar expressions
+ExecOption: Codegen Disabled for const scalar expressions, Codegen Enabled
====
---- QUERY
# Test insert statement with values having const scalar expressions.
@@ -43,7 +43,7 @@ insert into test_values_codegen values
00:UNION
constant-operands=2
#UNION_NODE
-ExecOption: Codegen Enabled, Codegen Disabled for const scalar expressions
+ExecOption: Codegen Disabled for const scalar expressions, Codegen Enabled
====
---- QUERY
# Test the result of above inserts with codegen disabled.
diff --git a/tests/common/test_result_verifier.py b/tests/common/test_result_verifier.py
index 02e3b88eb..0a08fc264 100644
--- a/tests/common/test_result_verifier.py
+++ b/tests/common/test_result_verifier.py
@@ -607,9 +607,14 @@ def verify_runtime_profile(expected, actual, update_section=False):
expected_lines = remove_comments(expected).splitlines()
matched = [False] * len(expected_lines)
expected_regexes = []
+ unexpected_regexes = []
+ unexpected_matched_lines = []
expected_aggregations = []
for expected_line in expected_lines:
- expected_regexes.append(try_compile_regex(expected_line))
+ negate_regex = expected_line and expected_line[0] == '!'
+ regex = try_compile_regex(expected_line[1:] if negate_regex else expected_line)
+ unexpected_regexes.append(regex if negate_regex else None)
+ expected_regexes.append(regex if not negate_regex else None)
expected_aggregations.append(try_compile_aggregation(expected_line))
# Check the expected and actual rows pairwise.
@@ -621,6 +626,10 @@ def verify_runtime_profile(expected, actual, update_section=False):
elif expected_aggregations[i] is not None:
# Aggregations are enforced separately
match = True
+ elif unexpected_regexes[i] is not None:
+ if unexpected_regexes[i].match(line):
+ unexpected_matched_lines.append(line)
+ match = False
else:
match = expected_lines[i].strip() == line.strip()
if match:
@@ -629,11 +638,14 @@ def verify_runtime_profile(expected, actual, update_section=False):
unmatched_lines = []
for i in xrange(len(expected_lines)):
- if not matched[i]:
+ if not matched[i] and unexpected_regexes[i] is None:
unmatched_lines.append(expected_lines[i])
assert len(unmatched_lines) == 0, ("Did not find matches for lines in runtime profile:"
"\nEXPECTED LINES:\n%s\n\nACTUAL PROFILE:\n%s" % ('\n'.join(unmatched_lines),
actual))
+ assert len(unexpected_matched_lines) == 0, ("Found unexpected matches in "
+ "runtime profile:\n%s\n\nACTUAL PROFILE:\n%s"
+ % ('\n'.join(unexpected_matched_lines), actual))
updated_aggregations = []
# Compute the aggregations and check against values