You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2019/05/01 21:01:25 UTC
[impala] branch master updated (eb97c74 -> f07be8d)
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from eb97c74 IMPALA-8293 (Part 2): Add support for Ranger cache invalidation
new 436a70e IMPALA-5031: signed overflow in TimestampValue
new f07be8d IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/runtime/timestamp-value.inline.h | 4 ++-
docs/topics/impala_explain_plan.xml | 43 +++++++++++++++++++++++----------
2 files changed, 33 insertions(+), 14 deletions(-)
[impala] 01/02: IMPALA-5031: signed overflow in TimestampValue
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 436a70e604d92ba17d11680dfdccc95988257a98
Author: Jim Apple <jb...@apache.org>
AuthorDate: Sat Nov 10 17:39:23 2018 -0800
IMPALA-5031: signed overflow in TimestampValue
The standard says that overflow for signed arithmetic operations is
undefined behavior; see [expr]:
If during the evaluation of an expression, the result is not
mathematically defined or not in the range of representable values
for its type, the behavior is undefined.
This patch fixes a signed overflow with the folowing backtrace
(uninteresting parts elided):
runtime/timestamp-value.inline.h:67:13: runtime error: signed integer overflow: -9223372036854775808 + -9223372037 cannot be represented in type 'long'
#0 TimestampValue::FromUnixTimeNanos(long, long, cctz::time_zone const&) runtime/timestamp-value.inline.h:67:13
#1 TimestampValue::FromSubsecondUnixTime(double, cctz::time_zone const&) runtime/timestamp-value.inline.h:62:10
#2 CastFunctions::CastToTimestampVal(impala_udf::FunctionContext*, impala_udf::FloatVal const&) exprs/cast-functions-ir.cc:248:172
#3 impala_udf::TimestampVal ScalarFnCall::InterpretEval<impala_udf::TimestampVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:208
#4 ScalarFnCall::GetTimestampVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:608:44
#5 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:314:41
#6 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#8 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#9 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#10 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#11 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
This was seen in the backend test ExprTest.CastExprs.
Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41
Reviewed-on: http://gerrit.cloudera.org:8080/11919
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/runtime/timestamp-value.inline.h | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/be/src/runtime/timestamp-value.inline.h b/be/src/runtime/timestamp-value.inline.h
index 2e2adac..1aebd78 100644
--- a/be/src/runtime/timestamp-value.inline.h
+++ b/be/src/runtime/timestamp-value.inline.h
@@ -28,6 +28,7 @@
#include "exprs/timezone_db.h"
#include "kudu/util/int128.h"
#include "gutil/walltime.h"
+#include "util/arithmetic-util.h"
namespace impala {
@@ -70,7 +71,8 @@ inline TimestampValue TimestampValue::UtcFromUnixTimeLimitedRangeNanos(
inline TimestampValue TimestampValue::FromUnixTimeNanos(time_t unix_time, int64_t nanos,
const Timezone& local_tz) {
- unix_time += SplitTime<NANOS_PER_SEC>(&nanos);
+ unix_time =
+ ArithmeticUtil::AsUnsigned<std::plus>(unix_time, SplitTime<NANOS_PER_SEC>(&nanos));
TimestampValue result = FromUnixTime(unix_time, local_tz);
// 'nanos' is guaranteed to be between [0,NANOS_PER_SEC) at this point, so the
// next addition cannot change the day or step to a different timezone.
[impala] 02/02: IMPALA-7892 IMPALA-8416: [DOCS] Described the new
network and disk info in query profiles
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit f07be8dfd802d41ee5a4011d17f4aad4d7208d65
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Fri Apr 12 18:00:52 2019 -0700
IMPALA-7892 IMPALA-8416: [DOCS] Described the new network and disk info in query profiles
- HostDiskReadThroughput
- HostDiskWriteThroughput
- HostNetworkRx
- HostNetworkTx
Change-Id: I25b128bc23f418347b400ca9e694d9d591935592
Reviewed-on: http://gerrit.cloudera.org:8080/13006
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Lars Volker <lv...@cloudera.com>
---
docs/topics/impala_explain_plan.xml | 43 ++++++++++++++++++++++++++-----------
1 file changed, 30 insertions(+), 13 deletions(-)
diff --git a/docs/topics/impala_explain_plan.xml b/docs/topics/impala_explain_plan.xml
index d983c21..b8f3168 100644
--- a/docs/topics/impala_explain_plan.xml
+++ b/docs/topics/impala_explain_plan.xml
@@ -189,16 +189,18 @@ under the License.
<conbody>
- <p>
- The <codeph>PROFILE</codeph> statement, available in the <cmdname>impala-shell</cmdname> interpreter,
- produces a detailed low-level report showing how the most recent query was executed. Unlike the
- <codeph>EXPLAIN</codeph> plan described in <xref href="#perf_explain"/>, this information is only available
- after the query has finished. It shows physical details such as the number of bytes read, maximum memory
- usage, and so on for each node. You can use this information to determine if the query is I/O-bound or
- CPU-bound, whether some network condition is imposing a bottleneck, whether a slowdown is affecting some
- nodes but not others, and to check that recommended configuration settings such as short-circuit local
- reads are in effect.
- </p>
+ <p> The <codeph>PROFILE</codeph> command, available in the
+ <cmdname>impala-shell</cmdname> interpreter, produces a detailed
+ low-level report showing how the most recent query was executed. Unlike
+ the <codeph>EXPLAIN</codeph> plan described in <xref
+ href="#perf_explain"/>, this information is only available after the
+ query has finished. It shows physical details such as the number of
+ bytes read, maximum memory usage, and so on for each node. You can use
+ this information to determine if the query is I/O-bound or CPU-bound,
+ whether some network condition is imposing a bottleneck, whether a
+ slowdown is affecting some nodes but not others, and to check that
+ recommended configuration settings such as short-circuit local reads are
+ in effect. </p>
<p rev="">
By default, time values in the profile output reflect the wall-clock time taken by an operation.
@@ -223,14 +225,29 @@ under the License.
section includes the following metrics that can be controlled by the
<codeph><xref
href="impala_resource_trace_ratio.xml#resource_trace_ratio"
- >RESOURCE_TRACE_RATIO</xref></codeph> query option. The host CPU
- usage metrics (user, system, and IO wait time) are already in the
- section.</p>
+ >RESOURCE_TRACE_RATIO</xref></codeph> query option.</p>
<ul>
+ <li>For each host that participates in the query execution it adds the
+ read and write bandwidth across all disks. This includes all data read
+ or written by the host as part of the execution of a query (spilling),
+ by the HDFS data node, and by other processes running on the same
+ system.</li>
<li><codeph>CpuIoWaitPercentage</codeph>
</li>
<li><codeph>CpuSysPercentage</codeph></li>
<li><codeph>CpuUserPercentage</codeph></li>
+ <li><codeph>HostDiskReadThroughput</codeph>: All data read by the host
+ as part of the execution of this query (spilling), by the HDFS data
+ node, and by other processes running on the same system.</li>
+ <li><codeph>HostDiskWriteThroughput</codeph>: All data written by the
+ host as part of the execution of this query (spilling), by the HDFS
+ data node, and by other processes running on the same system.</li>
+ <li><codeph>HostNetworkRx</codeph>: All data received by the host as
+ part of the execution of this query, other queries, and other
+ processes running on the same system. </li>
+ <li><codeph>HostNetworkTx</codeph>: All data transmitted by the host as
+ part of the execution of this query, other queries, and other
+ processes running on the same system. </li>
</ul>
<!--AR 3/11/2019 The below example is out dated and does not add much value. Hiding it until this doc gets refactored.-->