You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jim Apple (Code Review)" <ge...@cloudera.org> on 2019/05/25 21:52:39 UTC
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13436
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
IMPALA-5031: NULL is undefined in memcpy and memcmp
This patch fixes UBSAN "null pointer passed as argument" errors in the
end-to-end tests. These are undefined behavior according to "7.1.4 Use
of library functions" in the C99 standard (which is included in C++14
in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces are:
runtime/sorter.cc:575:18: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Sorter::Run::CopyVarLenData(vector<StringValue*> const&,
unsigned char*) runtime/sorter.cc:575:5
#1 Status Sorter::Run::AddBatchInternal<true, true>(RowBatch*,
int, int*) runtime/sorter.cc:232:11
#2 Sorter::Run::AddInputBatch(RowBatch*, int, int*)
runtime/sorter.cc:660:12
#3 Sorter::AddBatchNoSpill(RowBatch*, int, int*)
runtime/sorter.cc:882:58
#4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45
#5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54
#6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43
runtime/tuple.cc:105:25: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:105:5
#1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*)
runtime/tuple.cc:94:35
#2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:85:3
#3 KrpcDataStreamSender::Channel::AddRow(TupleRow*)
runtime/krpc-data-stream-sender.cc:509:43
#4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*)
runtime/krpc-data-stream-sender.cc:846
#5 (<unknown module>)
runtime/tuple.cc:146:19: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*,
bool) runtime/tuple.cc:146:5
#1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool)
runtime/tuple.cc:135:35
#2 RowBatch::SerializeInternal(long, FixedSizeHashTable<Tuple*,
int>*, vector<int>*, string*) runtime/row-batch.cc:392:14
#3 RowBatch::Serialize(bool, vector<int>*, string*, long*, bool*)
runtime/row-batch.cc:290:45
#4 RowBatch::Serialize(OutboundRowBatch*)
runtime/row-batch.cc:259:43
#5 KrpcDataStreamSender::SerializeBatch(RowBatch*,
OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50
#6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*)
runtime/krpc-data-stream-sender.cc:870:45
runtime/tuple.h:106:12: runtime error: null pointer passed as
argument 1, which is declared to never be null
/usr/include/string.h:62:79: note: nonnull attribute specified
here
#0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5
#1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*)
exec/hdfs-scanner.h:512:14
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:742:7
#3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#4 OrcStructReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:322:52
#5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*)
const exec/orc-column-readers.cc:473:52
#6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:743:60
#7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#8 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:52
#9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:58
#10 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:45
#11 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21
#13 HdfsScanNode::ProcessSplit(vector<FilterContext> const&,
MemPool*, io::ScanRange*, long*) exec/hdfs-scan-node.cc:514:21
#14 HdfsScanNode::ScannerThread(bool, long)
exec/hdfs-scan-node.cc:415:7
#15 HdfsScanNode::ThreadTokenAvailableCb(ThreadResourcePool*)
::$_0::operator()() const exec/hdfs-scan-node.cc:337:13
Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
---
M be/src/runtime/sorter.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
3 files changed, 7 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13436/1
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 2: Code-Review+2
Seems good once you fix the one bot comment.
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 May 2019 17:29:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 4
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Jun 2019 20:05:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
IMPALA-5031: NULL is undefined in memcpy and memcmp
This patch fixes UBSAN "null pointer passed as argument" errors in the
end-to-end tests. These are undefined behavior according to "7.1.4 Use
of library functions" in the C99 standard (which is included in C++14
in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces are:
runtime/sorter.cc:575:18: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Sorter::Run::CopyVarLenData(vector<StringValue*> const&,
unsigned char*) runtime/sorter.cc:575:5
#1 Status Sorter::Run::AddBatchInternal<true, true>(RowBatch*,
int, int*) runtime/sorter.cc:232:11
#2 Sorter::Run::AddInputBatch(RowBatch*, int, int*)
runtime/sorter.cc:660:12
#3 Sorter::AddBatchNoSpill(RowBatch*, int, int*)
runtime/sorter.cc:882:58
#4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45
#5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54
#6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43
runtime/tuple.cc:105:25: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:105:5
#1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*)
runtime/tuple.cc:94:35
#2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:85:3
#3 KrpcDataStreamSender::Channel::AddRow(TupleRow*)
runtime/krpc-data-stream-sender.cc:509:43
#4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*)
runtime/krpc-data-stream-sender.cc:846
#5 (<unknown module>)
runtime/tuple.cc:146:19: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*,
bool) runtime/tuple.cc:146:5
#1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool)
runtime/tuple.cc:135:35
#2 RowBatch::SerializeInternal(long, FixedSizeHashTable<Tuple*,
int>*, vector<int>*, string*) runtime/row-batch.cc:392:14
#3 RowBatch::Serialize(bool, vector<int>*, string*, long*, bool*)
runtime/row-batch.cc:290:45
#4 RowBatch::Serialize(OutboundRowBatch*)
runtime/row-batch.cc:259:43
#5 KrpcDataStreamSender::SerializeBatch(RowBatch*,
OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50
#6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*)
runtime/krpc-data-stream-sender.cc:870:45
runtime/tuple.h:106:12: runtime error: null pointer passed as
argument 1, which is declared to never be null
/usr/include/string.h:62:79: note: nonnull attribute specified
here
#0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5
#1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*)
exec/hdfs-scanner.h:512:14
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:742:7
#3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#4 OrcStructReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:322:52
#5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*)
const exec/orc-column-readers.cc:473:52
#6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:743:60
#7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#8 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:52
#9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:58
#10 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:45
#11 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21
#13 HdfsScanNode::ProcessSplit(vector<FilterContext> const&,
MemPool*, io::ScanRange*, long*) exec/hdfs-scan-node.cc:514:21
#14 HdfsScanNode::ScannerThread(bool, long)
exec/hdfs-scan-node.cc:415:7
#15 HdfsScanNode::ThreadTokenAvailableCb(ThreadResourcePool*)
::$_0::operator()() const exec/hdfs-scan-node.cc:337:13
runtime/collection-value-builder.h:75:25: runtime error: null
pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:28: note: nonnull attribute specified
here
#0 CollectionValueBuilder::GetFreeMemory(Tuple**, int*)
runtime/collection-value-builder.h:75:9
#1 HdfsScanner::GetCollectionMemory(CollectionValueBuilder*,
MemPool**, Tuple**, TupleRow**, long*)
exec/hdfs-scanner.cc:194:3
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:733:9
#3 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#4 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#5 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#6 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:5
#7 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:5
#8 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:5
#9 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#10 HdfsScanner::GetNext(RowBatch*) exec/hdfs-scanner.h:133:12
#11 HdfsScanNodeMt::GetNext(RuntimeState*, RowBatch*, bool*)
exec/hdfs-scan-node-mt.cc:106:29
#12 SubplanNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/subplan-node.cc:129:7
#13 AggregationNode::Open(RuntimeState*)
exec/aggregation-node.cc:67:5
Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Reviewed-on: http://gerrit.cloudera.org:8080/13436
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/collection-value-builder.h
M be/src/runtime/sorter.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
4 files changed, 10 insertions(+), 5 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 5
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13436/2/be/src/runtime/tuple.h
File be/src/runtime/tuple.h:
http://gerrit.cloudera.org:8080/#/c/13436/2/be/src/runtime/tuple.h@107
PS2, Line 107: Ubsan::MemSet(reinterpret_cast<uint8_t*>(this) + null_bytes_offset, 0, num_null_bytes);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 26 May 2019 01:33:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 3: Code-Review+2
Carry Tim's +2
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Jun 2019 20:05:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13436/1/be/src/runtime/tuple.h
File be/src/runtime/tuple.h:
http://gerrit.cloudera.org:8080/#/c/13436/1/be/src/runtime/tuple.h@107
PS1, Line 107: Ubsan::MemSet(reinterpret_cast<uint8_t*>(this) + null_bytes_offset, 0, num_null_bytes);
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 25 May 2019 21:53:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3395/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 26 May 2019 02:13:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13436
to look at the new patch set (#2).
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
IMPALA-5031: NULL is undefined in memcpy and memcmp
This patch fixes UBSAN "null pointer passed as argument" errors in the
end-to-end tests. These are undefined behavior according to "7.1.4 Use
of library functions" in the C99 standard (which is included in C++14
in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces are:
runtime/sorter.cc:575:18: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Sorter::Run::CopyVarLenData(vector<StringValue*> const&,
unsigned char*) runtime/sorter.cc:575:5
#1 Status Sorter::Run::AddBatchInternal<true, true>(RowBatch*,
int, int*) runtime/sorter.cc:232:11
#2 Sorter::Run::AddInputBatch(RowBatch*, int, int*)
runtime/sorter.cc:660:12
#3 Sorter::AddBatchNoSpill(RowBatch*, int, int*)
runtime/sorter.cc:882:58
#4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45
#5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54
#6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43
runtime/tuple.cc:105:25: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:105:5
#1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*)
runtime/tuple.cc:94:35
#2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:85:3
#3 KrpcDataStreamSender::Channel::AddRow(TupleRow*)
runtime/krpc-data-stream-sender.cc:509:43
#4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*)
runtime/krpc-data-stream-sender.cc:846
#5 (<unknown module>)
runtime/tuple.cc:146:19: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*,
bool) runtime/tuple.cc:146:5
#1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool)
runtime/tuple.cc:135:35
#2 RowBatch::SerializeInternal(long, FixedSizeHashTable<Tuple*,
int>*, vector<int>*, string*) runtime/row-batch.cc:392:14
#3 RowBatch::Serialize(bool, vector<int>*, string*, long*, bool*)
runtime/row-batch.cc:290:45
#4 RowBatch::Serialize(OutboundRowBatch*)
runtime/row-batch.cc:259:43
#5 KrpcDataStreamSender::SerializeBatch(RowBatch*,
OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50
#6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*)
runtime/krpc-data-stream-sender.cc:870:45
runtime/tuple.h:106:12: runtime error: null pointer passed as
argument 1, which is declared to never be null
/usr/include/string.h:62:79: note: nonnull attribute specified
here
#0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5
#1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*)
exec/hdfs-scanner.h:512:14
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:742:7
#3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#4 OrcStructReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:322:52
#5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*)
const exec/orc-column-readers.cc:473:52
#6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:743:60
#7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#8 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:52
#9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:58
#10 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:45
#11 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21
#13 HdfsScanNode::ProcessSplit(vector<FilterContext> const&,
MemPool*, io::ScanRange*, long*) exec/hdfs-scan-node.cc:514:21
#14 HdfsScanNode::ScannerThread(bool, long)
exec/hdfs-scan-node.cc:415:7
#15 HdfsScanNode::ThreadTokenAvailableCb(ThreadResourcePool*)
::$_0::operator()() const exec/hdfs-scan-node.cc:337:13
runtime/collection-value-builder.h:75:25: runtime error: null
pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:28: note: nonnull attribute specified
here
#0 CollectionValueBuilder::GetFreeMemory(Tuple**, int*)
runtime/collection-value-builder.h:75:9
#1 HdfsScanner::GetCollectionMemory(CollectionValueBuilder*,
MemPool**, Tuple**, TupleRow**, long*)
exec/hdfs-scanner.cc:194:3
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:733:9
#3 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#4 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#5 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#6 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:5
#7 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:5
#8 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:5
#9 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#10 HdfsScanner::GetNext(RowBatch*) exec/hdfs-scanner.h:133:12
#11 HdfsScanNodeMt::GetNext(RuntimeState*, RowBatch*, bool*)
exec/hdfs-scan-node-mt.cc:106:29
#12 SubplanNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/subplan-node.cc:129:7
#13 AggregationNode::Open(RuntimeState*)
exec/aggregation-node.cc:67:5
Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
---
M be/src/runtime/collection-value-builder.h
M be/src/runtime/sorter.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
4 files changed, 9 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13436/2
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 2
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3477/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Jun 2019 20:44:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 25 May 2019 22:32:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4383/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 4
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Jun 2019 20:05:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13436
to look at the new patch set (#3).
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
IMPALA-5031: NULL is undefined in memcpy and memcmp
This patch fixes UBSAN "null pointer passed as argument" errors in the
end-to-end tests. These are undefined behavior according to "7.1.4 Use
of library functions" in the C99 standard (which is included in C++14
in section [intro.refs]):
If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the
address space of the program, or a null pointer, or a pointer to
non-modifiable storage when the corresponding parameter is not
const-qualified) or a type (after promotion) not expected by a
function with variable number of arguments, the behavior is
undefined.
The interesting parts of the backtraces are:
runtime/sorter.cc:575:18: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Sorter::Run::CopyVarLenData(vector<StringValue*> const&,
unsigned char*) runtime/sorter.cc:575:5
#1 Status Sorter::Run::AddBatchInternal<true, true>(RowBatch*,
int, int*) runtime/sorter.cc:232:11
#2 Sorter::Run::AddInputBatch(RowBatch*, int, int*)
runtime/sorter.cc:660:12
#3 Sorter::AddBatchNoSpill(RowBatch*, int, int*)
runtime/sorter.cc:882:58
#4 Sorter::AddBatch(RowBatch*) runtime/sorter.cc:862:45
#5 SortNode::SortInput(RuntimeState*) exec/sort-node.cc:177:54
#6 SortNode::Open(RuntimeState*) exec/sort-node.cc:90:43
runtime/tuple.cc:105:25: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:105:5
#1 Tuple::DeepCopy(Tuple*, TupleDescriptor const&, MemPool*)
runtime/tuple.cc:94:35
#2 Tuple::DeepCopy(TupleDescriptor const&, MemPool*)
runtime/tuple.cc:85:3
#3 KrpcDataStreamSender::Channel::AddRow(TupleRow*)
runtime/krpc-data-stream-sender.cc:509:43
#4 KrpcDataStreamSender::AddRowToChannel(int, TupleRow*)
runtime/krpc-data-stream-sender.cc:846
#5 (<unknown module>)
runtime/tuple.cc:146:19: runtime error: null pointer passed as
argument 2, which is declared to never be null
/usr/include/string.h:43:45: note: nonnull attribute specified
here
#0 Tuple::DeepCopyVarlenData(TupleDescriptor const&, char**, int*,
bool) runtime/tuple.cc:146:5
#1 Tuple::DeepCopy(TupleDescriptor const&, char**, int*, bool)
runtime/tuple.cc:135:35
#2 RowBatch::SerializeInternal(long, FixedSizeHashTable<Tuple*,
int>*, vector<int>*, string*) runtime/row-batch.cc:392:14
#3 RowBatch::Serialize(bool, vector<int>*, string*, long*, bool*)
runtime/row-batch.cc:290:45
#4 RowBatch::Serialize(OutboundRowBatch*)
runtime/row-batch.cc:259:43
#5 KrpcDataStreamSender::SerializeBatch(RowBatch*,
OutboundRowBatch*, int) runtime/krpc-data-stream-sender.cc:955:50
#6 KrpcDataStreamSender::Send(RuntimeState*, RowBatch*)
runtime/krpc-data-stream-sender.cc:870:45
runtime/tuple.h:106:12: runtime error: null pointer passed as
argument 1, which is declared to never be null
/usr/include/string.h:62:79: note: nonnull attribute specified
here
#0 Tuple::ClearNullBits(int, int) runtime/tuple.h:106:5
#1 HdfsScanner::InitTuple(TupleDescriptor const*, Tuple*, Tuple*)
exec/hdfs-scanner.h:512:14
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:742:7
#3 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#4 OrcStructReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:322:52
#5 OrcListReader::ReadChildrenValue(int, int, Tuple*, MemPool*)
const exec/orc-column-readers.cc:473:52
#6 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:743:60
#7 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#8 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:52
#9 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:58
#10 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:45
#11 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#12 HdfsOrcScanner::ProcessSplit() exec/hdfs-orc-scanner.cc:426:21
#13 HdfsScanNode::ProcessSplit(vector<FilterContext> const&,
MemPool*, io::ScanRange*, long*) exec/hdfs-scan-node.cc:514:21
#14 HdfsScanNode::ScannerThread(bool, long)
exec/hdfs-scan-node.cc:415:7
#15 HdfsScanNode::ThreadTokenAvailableCb(ThreadResourcePool*)
::$_0::operator()() const exec/hdfs-scan-node.cc:337:13
runtime/collection-value-builder.h:75:25: runtime error: null
pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:28: note: nonnull attribute specified
here
#0 CollectionValueBuilder::GetFreeMemory(Tuple**, int*)
runtime/collection-value-builder.h:75:9
#1 HdfsScanner::GetCollectionMemory(CollectionValueBuilder*,
MemPool**, Tuple**, TupleRow**, long*)
exec/hdfs-scanner.cc:194:3
#2 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:733:9
#3 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#4 HdfsOrcScanner::AssembleCollection(OrcComplexColumnReader
const&, int, CollectionValueBuilder*)
exec/hdfs-orc-scanner.cc:710:7
#5 OrcCollectionReader::ReadValue(int, Tuple*, MemPool*)
exec/orc-column-readers.cc:375:20
#6 OrcStructReader::TransferTuple(Tuple*, MemPool*)
exec/orc-column-readers.cc:346:5
#7 HdfsOrcScanner::TransferTuples(OrcComplexColumnReader*,
RowBatch*) exec/hdfs-orc-scanner.cc:669:5
#8 HdfsOrcScanner::AssembleRows(RowBatch*)
exec/hdfs-orc-scanner.cc:629:5
#9 HdfsOrcScanner::GetNextInternal(RowBatch*)
exec/hdfs-orc-scanner.cc:507:19
#10 HdfsScanner::GetNext(RowBatch*) exec/hdfs-scanner.h:133:12
#11 HdfsScanNodeMt::GetNext(RuntimeState*, RowBatch*, bool*)
exec/hdfs-scan-node-mt.cc:106:29
#12 SubplanNode::GetNext(RuntimeState*, RowBatch*, bool*)
exec/subplan-node.cc:129:7
#13 AggregationNode::Open(RuntimeState*)
exec/aggregation-node.cc:67:5
Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
---
M be/src/runtime/collection-value-builder.h
M be/src/runtime/sorter.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
4 files changed, 10 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13436/3
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 )
Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5
Gerrit-Change-Number: 13436
Gerrit-PatchSet: 4
Gerrit-Owner: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 02 Jun 2019 01:40:40 +0000
Gerrit-HasComments: No