You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jim Apple (Code Review)" <ge...@cloudera.org> on 2018/09/30 03:02:30 UTC
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11545
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
IMPALA-5031: null ptr errors in C calls in BE tests
This patch fixes all remaining UBSAN "null pointer passed as argument"
errors in the backend tests. These are undefined behavior according to
"7.1.4 Use of library functions" in the C99 standard (which is
included in C++14 in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces for the errors fixed in this
patch are below:
exprs/string-functions-ir.cc:311:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::Replace(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::StringVal const&) exprs/string-functions-ir.cc:311:5
#1 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:868:15: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:868:3
#1 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:270
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:871:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:871:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:873:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:873:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
runtime/raw-value.cc:159:27: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 RawValue::Write(void const*, void*, ColumnType const&, MemPool*) runtime/raw-value.cc:159:9
#1 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:225:7
#2 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#3 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#4 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#5 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#6 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#7 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#8 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#9 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#16 thread_proxy (exprs/expr-test+0x55ca939)
udf/udf.cc:521:24: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 impala_udf::StringVal::CopyFrom(impala_udf::FunctionContext*, unsigned char const*, unsigned long) udf/udf.cc:521:5
#1 AnyValUtil::FromBuffer(impala_udf::FunctionContext*, char const*, int) exprs/anyval-util.h:241:12
#2 StringFunctions::RegexpExtract(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::BigIntVal const&) exprs/string-functions-ir.cc:726:10
#3 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#4 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#5 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#6 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#8 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#9 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#10 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#11 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#12 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#13 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#14 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#15 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#22 thread_proxy (exprs/expr-test+0x55ca939)
util/coding-util-test.cc:45:10: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 TestUrl(string const&, string const&, bool) util/coding-util-test.cc:45:3
#1 UrlCodingTest_BlankString_Test::TestBody() util/coding-util-test.cc:88:3
#2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/coding-util-test+0x6630f42)
#8 main util/coding-util-test.cc:123:192
util/decompress-test.cc:126:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:126:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:148:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:148:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:269:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompressNoOutputAllocated(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:269:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:71:7
#2 DecompressorTest_LZ4_Test::TestBody() util/decompress-test.cc:381:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:221:329: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::StreamingDecompress(Codec*, long, unsigned char*, long, unsigned char*, bool, long*) util/decompress-test.cc:221:322
#1 DecompressorTest::CompressAndStreamingDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:245:35
#2 DecompressorTest::RunTestStreaming(THdfsCompression::type) util/decompress-test.cc:104:5
#3 DecompressorTest_Gzip_Test::TestBody() util/decompress-test.cc:386:3
#4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#10 main util/decompress-test.cc:479:47
util/streaming-sampler.h:55:22: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StreamingSampler<long, 64>::StreamingSampler(int, vector<long> const&) util/streaming-sampler.h:55:5
#1 RuntimeProfile::TimeSeriesCounter::TimeSeriesCounter(string const&, TUnit::type, int, vector<long> const&) util/runtime-profile-counters.h:401:53
#2 RuntimeProfile::Update(vector<TRuntimeProfileNode> const&, int*) util/runtime-profile.cc:310:28
#3 RuntimeProfile::Update(TRuntimeProfileTree const&) util/runtime-profile.cc:245:3
#4 Coordinator::BackendState::InstanceStats::Update(TFragmentInstanceExecStatus const&, Coordinator::ExecSummary*, ProgressUpdater*) runtime/coordinator-backend-state.cc:473:13
#5 Coordinator::BackendState::ApplyExecStatusReport(TReportExecStatusParams const&, Coordinator::ExecSummary*, ProgressUpdater*) runtime/coordinator-backend-state.cc:286:21
#6 Coordinator::UpdateBackendExecStatus(TReportExecStatusParams const&) runtime/coordinator.cc:678:22
#7 ClientRequestState::UpdateBackendExecStatus(TReportExecStatusParams const&) service/client-request-state.cc:1253:18
#8 ImpalaServer::ReportExecStatus(TReportExecStatusResult&, TReportExecStatusParams const&) service/impala-server.cc:1343:18
#9 ImpalaInternalService::ReportExecStatus(TReportExecStatusResult&, TReportExecStatusParams const&) service/impala-internal-service.cc:87:19
#24 thread_proxy (exprs/expr-test+0x55ca939)
Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
---
M be/src/exprs/string-functions-ir.cc
M be/src/runtime/raw-value.cc
M be/src/udf/udf.cc
M be/src/util/coding-util-test.cc
M be/src/util/decompress-test.cc
M be/src/util/streaming-sampler.h
M be/src/util/ubsan.h
7 files changed, 27 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/11545/1
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc
File be/src/udf/udf.cc:
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc@521
PS1, Line 521: std::copy(buf, buf + len, result.ptr);
Does this compile to the same thing? This function is potentially a little perf-sensitive.
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Oct 2018 18:48:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1: Code-Review+2
I think probably only in certain string operations, but not sure that it's really that much on the critical path. Seems ok...
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Oct 2018 16:36:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc
File be/src/udf/udf.cc:
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc@521
PS1, Line 521: std::copy(buf, buf + len, result.ptr);
> Does this compile to the same thing? This function is potentially a little
I haven't checked, but it very well might not, or it might now but then stop doing so later.
I don't want to use ubsan.h, since it pulls in glog. Would you be OK with another LIKELY condition that only triggers when buf != nullptr?
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Oct 2018 18:56:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3263/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Oct 2018 16:42:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Oct 2018 16:42:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Oct 2018 20:24:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
IMPALA-5031: null ptr errors in C calls in BE tests
This patch fixes all remaining UBSAN "null pointer passed as argument"
errors in the backend tests. These are undefined behavior according to
"7.1.4 Use of library functions" in the C99 standard (which is
included in C++14 in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces for the errors fixed in this
patch are below:
exprs/string-functions-ir.cc:311:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::Replace(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::StringVal const&) exprs/string-functions-ir.cc:311:5
#1 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:868:15: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:868:3
#1 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:270
#2 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#3 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#4 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#5 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#7 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#8 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#9 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#10 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#11 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#12 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#13 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#20 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:871:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:871:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
exprs/string-functions-ir.cc:873:17: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StringFunctions::ConcatWs(impala_udf::FunctionContext*, impala_udf::StringVal const&, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:873:5
#1 StringFunctions::Concat(impala_udf::FunctionContext*, int, impala_udf::StringVal const*) exprs/string-functions-ir.cc:843:10
#2 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:510:95
#3 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#4 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#5 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#6 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#8 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#9 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#10 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#11 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#12 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#13 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#14 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#21 thread_proxy (exprs/expr-test+0x55ca939)
runtime/raw-value.cc:159:27: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 RawValue::Write(void const*, void*, ColumnType const&, MemPool*) runtime/raw-value.cc:159:9
#1 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:225:7
#2 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#3 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#4 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#5 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#6 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#7 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#8 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#9 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#16 thread_proxy (exprs/expr-test+0x55ca939)
udf/udf.cc:521:24: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 impala_udf::StringVal::CopyFrom(impala_udf::FunctionContext*, unsigned char const*, unsigned long) udf/udf.cc:521:5
#1 AnyValUtil::FromBuffer(impala_udf::FunctionContext*, char const*, int) exprs/anyval-util.h:241:12
#2 StringFunctions::RegexpExtract(impala_udf::FunctionContext*, impala_udf::StringVal const&, impala_udf::StringVal const&, impala_udf::BigIntVal const&) exprs/string-functions-ir.cc:726:10
#3 impala_udf::StringVal ScalarFnCall::InterpretEval<impala_udf::StringVal>(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:580
#4 ScalarFnCall::GetStringVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:599:44
#5 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:299:38
#6 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10
#7 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27
#8 void Tuple::MaterializeExprs<false, false>(TupleRow*, TupleDescriptor const&, vector<ScalarExprEvaluator*> const&, MemPool*, vector<StringValue*>*, int*) runtime/tuple.h:174:5
#9 UnionNode::MaterializeExprs(vector<ScalarExprEvaluator*> const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14
#10 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5
#11 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45
#12 FragmentInstanceState::ExecInternal() runtime/fragment-instance-state.cc:310:59
#13 FragmentInstanceState::Exec() runtime/fragment-instance-state.cc:95:14
#14 QueryState::ExecFInstance(FragmentInstanceState*) runtime/query-state.cc:488:24
#15 QueryState::StartFInstances()::$_0::operator()() const runtime/query-state.cc:416:35
#22 thread_proxy (exprs/expr-test+0x55ca939)
util/coding-util-test.cc:45:10: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 TestUrl(string const&, string const&, bool) util/coding-util-test.cc:45:3
#1 UrlCodingTest_BlankString_Test::TestBody() util/coding-util-test.cc:88:3
#2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/coding-util-test+0x6630f42)
#8 main util/coding-util-test.cc:123:192
util/decompress-test.cc:126:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:126:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:148:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:148:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:84:9
#2 DecompressorTest_Default_Test::TestBody() util/decompress-test.cc:373:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:269:261: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::CompressAndDecompressNoOutputAllocated(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:269:254
#1 DecompressorTest::RunTest(THdfsCompression::type) util/decompress-test.cc:71:7
#2 DecompressorTest_LZ4_Test::TestBody() util/decompress-test.cc:381:3
#3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#9 main util/decompress-test.cc:479:47
util/decompress-test.cc:221:329: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/string.h:66:58: note: nonnull attribute specified here
#0 DecompressorTest::StreamingDecompress(Codec*, long, unsigned char*, long, unsigned char*, bool, long*) util/decompress-test.cc:221:322
#1 DecompressorTest::CompressAndStreamingDecompress(Codec*, Codec*, long, unsigned char*) util/decompress-test.cc:245:35
#2 DecompressorTest::RunTestStreaming(THdfsCompression::type) util/decompress-test.cc:104:5
#3 DecompressorTest_Gzip_Test::TestBody() util/decompress-test.cc:386:3
#4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (util/decompress-test+0x6642bb2)
#10 main util/decompress-test.cc:479:47
util/streaming-sampler.h:55:22: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified here
#0 StreamingSampler<long, 64>::StreamingSampler(int, vector<long> const&) util/streaming-sampler.h:55:5
#1 RuntimeProfile::TimeSeriesCounter::TimeSeriesCounter(string const&, TUnit::type, int, vector<long> const&) util/runtime-profile-counters.h:401:53
#2 RuntimeProfile::Update(vector<TRuntimeProfileNode> const&, int*) util/runtime-profile.cc:310:28
#3 RuntimeProfile::Update(TRuntimeProfileTree const&) util/runtime-profile.cc:245:3
#4 Coordinator::BackendState::InstanceStats::Update(TFragmentInstanceExecStatus const&, Coordinator::ExecSummary*, ProgressUpdater*) runtime/coordinator-backend-state.cc:473:13
#5 Coordinator::BackendState::ApplyExecStatusReport(TReportExecStatusParams const&, Coordinator::ExecSummary*, ProgressUpdater*) runtime/coordinator-backend-state.cc:286:21
#6 Coordinator::UpdateBackendExecStatus(TReportExecStatusParams const&) runtime/coordinator.cc:678:22
#7 ClientRequestState::UpdateBackendExecStatus(TReportExecStatusParams const&) service/client-request-state.cc:1253:18
#8 ImpalaServer::ReportExecStatus(TReportExecStatusResult&, TReportExecStatusParams const&) service/impala-server.cc:1343:18
#9 ImpalaInternalService::ReportExecStatus(TReportExecStatusResult&, TReportExecStatusParams const&) service/impala-internal-service.cc:87:19
#24 thread_proxy (exprs/expr-test+0x55ca939)
Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Reviewed-on: http://gerrit.cloudera.org:8080/11545
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exprs/string-functions-ir.cc
M be/src/runtime/raw-value.cc
M be/src/udf/udf.cc
M be/src/util/coding-util-test.cc
M be/src/util/decompress-test.cc
M be/src/util/streaming-sampler.h
M be/src/util/ubsan.h
7 files changed, 27 insertions(+), 12 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/874/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 30 Sep 2018 03:38:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc
File be/src/udf/udf.cc:
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc@521
PS1, Line 521: std::copy(buf, buf + len, result.ptr);
> I tried this out on godbolt and it results in two extra redundant compariso
Here are some results from single_node_perf_run.py:
Report Generated on 2018-10-02
Run Description: "ac33c0c42e1e7cc898893b1ae1f69c13287d20a8 vs 13e6428fecfbc16f0a8ad36c89126167e9988b5e"
Cluster Name: UNKNOWN
Lab Run Info: UNKNOWN
Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE ()
Baseline Impala Version: impalad version 3.1.0-SNAPSHOT RELEASE (2018-09-28)
+----------+-----------------------+---------+------------+------------+----------------+
| Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) |
+----------+-----------------------+---------+------------+------------+----------------+
| TPCH(40) | parquet / none / none | 8.24 | -0.67% | 5.50 | -0.74% |
+----------+-----------------------+---------+------------+------------+----------------+
+----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
| Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval |
+----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
| TPCH(40) | TPCH-Q7 | parquet / none / none | 27.03 | 26.72 | +1.14% | 0.95% | 0.71% | 50 | +0.95% | 5.69 | 6.73 |
| TPCH(40) | TPCH-Q11 | parquet / none / none | 1.35 | 1.33 | +1.16% | 2.90% | 3.70% | 50 | +0.34% | 1.55 | 1.74 |
| TPCH(40) | TPCH-Q19 | parquet / none / none | 5.33 | 5.30 | +0.59% | 1.51% | 1.51% | 50 | +0.67% | 1.87 | 1.93 |
| TPCH(40) | TPCH-Q2 | parquet / none / none | 2.03 | 2.02 | +0.32% | 3.59% | 4.83% | 50 | +0.05% | 0.57 | 0.37 |
| TPCH(40) | TPCH-Q8 | parquet / none / none | 4.74 | 4.73 | +0.25% | 1.60% | 1.65% | 50 | +0.08% | 0.66 | 0.76 |
| TPCH(40) | TPCH-Q18 | parquet / none / none | 16.10 | 16.09 | +0.11% | 2.05% | 1.56% | 50 | -0.03% | -0.41 | 0.29 |
| TPCH(40) | TPCH-Q10 | parquet / none / none | 7.17 | 7.17 | +0.02% | 2.10% | 2.49% | 50 | +0.03% | 0.16 | 0.05 |
| TPCH(40) | TPCH-Q20 | parquet / none / none | 2.87 | 2.87 | -0.00% | 3.09% | 2.96% | 50 | -0.03% | -0.16 | -0.01 |
| TPCH(40) | TPCH-Q15 | parquet / none / none | 4.29 | 4.29 | -0.06% | 1.75% | 2.00% | 50 | -0.05% | -0.22 | -0.17 |
| TPCH(40) | TPCH-Q21 | parquet / none / none | 34.25 | 34.30 | -0.14% | 0.63% | 0.87% | 50 | -0.09% | -0.56 | -0.89 |
| TPCH(40) | TPCH-Q22 | parquet / none / none | 2.68 | 2.69 | -0.20% | 2.61% | 2.70% | 50 | -0.03% | -1.02 | -0.38 |
| TPCH(40) | TPCH-Q16 | parquet / none / none | 2.82 | 2.83 | -0.26% | 1.85% | 1.58% | 50 | -0.05% | -1.89 | -0.76 |
| TPCH(40) | TPCH-Q6 | parquet / none / none | 1.50 | 1.51 | -0.95% | 1.87% | 1.84% | 50 | -0.21% | -2.85 | -2.57 |
| TPCH(40) | TPCH-Q12 | parquet / none / none | 3.29 | 3.33 | -1.16% | 2.61% | 3.14% | 50 | -1.37% | -1.61 | -2.01 |
| TPCH(40) | TPCH-Q1 | parquet / none / none | 7.76 | 7.89 | -1.63% | 2.08% | 2.62% | 50 | -1.34% | -3.20 | -3.46 |
| TPCH(40) | TPCH-Q14 | parquet / none / none | 3.23 | 3.29 | -1.93% | 1.32% | 2.85% | 50 | -1.55% | -4.29 | -4.37 |
| TPCH(40) | TPCH-Q17 | parquet / none / none | 4.64 | 4.73 | -1.99% | 6.19% | 6.62% | 50 | -1.98% | -1.18 | -1.56 |
| TPCH(40) | TPCH-Q9 | parquet / none / none | 19.02 | 19.45 | -2.18% | 0.75% | 0.60% | 50 | -2.32% | -8.14 | -16.35 |
| TPCH(40) | TPCH-Q13 | parquet / none / none | 10.36 | 10.60 | -2.21% | 0.49% | 0.44% | 50 | -2.41% | -8.53 | -23.91 |
| TPCH(40) | TPCH-Q5 | parquet / none / none | 11.03 | 10.98 | +0.44% | 8.47% | * 23.56% * | 50 | -6.82% | -4.65 | 0.12 |
| TPCH(40) | TPCH-Q4 | parquet / none / none | 3.96 | 4.14 | -4.20% | 1.73% | 2.58% | 50 | -4.00% | -7.21 | -9.68 |
| TPCH(40) | TPCH-Q3 | parquet / none / none | 5.90 | 6.32 | I -6.61% | 1.12% | 4.35% | 50 | I -5.96% | -8.26 | -10.45 |
+----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+--------+
(I) Improvement: TPCH(40) TPCH-Q3 [parquet / none / none] (6.32s -> 5.90s [-6.61%])
+--------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+---------+-----------+
| Operator | % of Query | Avg | Base Avg | Delta(Avg) | StdDev(%) | Max | Base Max | Delta(Max) | #Hosts | #Rows | Est #Rows |
+--------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+---------+-----------+
| 05:AGGREGATE | 11.45% | 635.91ms | 631.23ms | +0.74% | 4.28% | 791.99ms | 771.99ms | +2.59% | 3 | 453.79K | 7.32M |
| 03:HASH JOIN | 66.03% | 3.67s | 4.00s | -8.35% | 1.27% | 4.00s | 4.79s | -16.53% | 3 | 1.47M | 25.40M |
| 07:EXCHANGE | 2.31% | 128.11ms | 127.29ms | +0.64% | 9.67% | 224.00ms | 220.00ms | +1.82% | 3 | 7.17M | 6.00M |
| 01:SCAN HDFS | 13.30% | 738.68ms | 769.30ms | -3.98% | 4.35% | 863.99ms | 859.99ms | +0.47% | 3 | 7.17M | 6.00M |
| 02:SCAN HDFS | 4.32% | 239.81ms | 263.40ms | -8.96% | 7.50% | 372.00ms | 375.99ms | -1.06% | 3 | 129.40M | 24.00M |
+--------------+------------+----------+----------+------------+-----------+----------+----------+------------+--------+---------+-----------+
Significant perf change detected
I didn't look into how to exercise this directly - I just assumed StringVal::CopyFrom is called frequently in common queries.
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Oct 2018 13:36:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: null ptr errors in C calls in BE tests
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11545 )
Change subject: IMPALA-5031: null ptr errors in C calls in BE tests
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc
File be/src/udf/udf.cc:
http://gerrit.cloudera.org:8080/#/c/11545/1/be/src/udf/udf.cc@521
PS1, Line 521: std::copy(buf, buf + len, result.ptr);
> I haven't checked, but it very well might not, or it might now but then sto
I tried this out on godbolt and it results in two extra redundant comparisons but otherwise the same call to memcpy(): https://godbolt.org/z/pbiUpm
So it would be good to confirm that this doesn't affect performance with a benchmark. I suspect not but who knows
--
To view, visit http://gerrit.cloudera.org:8080/11545
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I317ccc99549744a26d65f3e07242079faad0355a
Gerrit-Change-Number: 11545
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Oct 2018 19:22:04 +0000
Gerrit-HasComments: Yes