You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2018/02/27 04:35:46 UTC
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9459
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
IMPALA-6543: Limit RowBatch serialization size to INT_MAX
The serialization format of a row batch relies on
tuple offsets. In its current form, the tuple offsets
are int32s. This means that it is impossible to generate
a valid serialization of a row batch that is larger
than INT_MAX.
This changes RowBatch::SerializeInternal() to return an
error if trying to serialize a row batch larger than INT_MAX.
This prevents a DCHECK on debug builds when creating a row
larger than 2GB.
This also changes the compression logic in RowBatch::Serialize()
to avoid a DCHECK if LZ4 will not be able to compress the
row batch. Instead, it returns an error.
This modifies row-batch-serialize-test to verify behavior at
each of the limits. Specifically:
RowBatches up to size LZ4_MAX_INPUT_SIZE succeed.
RowBatches with size range [LZ4_MAX_INPUT_SIZE+1, INT_MAX]
fail on LZ4 compression.
RowBatches with size > INT_MAX fail with RowBatch too large.
Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
---
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M common/thrift/generate_error_codes.py
4 files changed, 131 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/9459/1
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/9459 )
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/9459/1/common/thrift/generate_error_codes.py
File common/thrift/generate_error_codes.py:
http://gerrit.cloudera.org:8080/#/c/9459/1/common/thrift/generate_error_codes.py@351
PS1, Line 351:
: ("PARQUET_BIT_PACKED_LEVELS", 115,
: "Can not read Parquet file $0 with deprecated BIT_PACKED encoding for rep or "
: "def levels. Support was removed in Impala 3.0 - see IMPALA-6077."),
Conflict: master had this error message and 2.x did not
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Feb 2018 04:38:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9459 )
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
IMPALA-6543: Limit RowBatch serialization size to INT_MAX
The serialization format of a row batch relies on
tuple offsets. In its current form, the tuple offsets
are int32s. This means that it is impossible to generate
a valid serialization of a row batch that is larger
than INT_MAX.
This changes RowBatch::SerializeInternal() to return an
error if trying to serialize a row batch larger than INT_MAX.
This prevents a DCHECK on debug builds when creating a row
larger than 2GB.
This also changes the compression logic in RowBatch::Serialize()
to avoid a DCHECK if LZ4 will not be able to compress the
row batch. Instead, it returns an error.
This modifies row-batch-serialize-test to verify behavior at
each of the limits. Specifically:
RowBatches up to size LZ4_MAX_INPUT_SIZE succeed.
RowBatches with size range [LZ4_MAX_INPUT_SIZE+1, INT_MAX]
fail on LZ4 compression.
RowBatches with size > INT_MAX fail with RowBatch too large.
Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Reviewed-on: http://gerrit.cloudera.org:8080/9459
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M common/thrift/generate_error_codes.py
4 files changed, 131 insertions(+), 12 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9459 )
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Feb 2018 19:46:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9459 )
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2023/
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Feb 2018 16:05:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6543: Limit RowBatch serialization size to INT MAX
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9459 )
Change subject: IMPALA-6543: Limit RowBatch serialization size to INT_MAX
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/9459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I1f93131facf47bd8d86c3d4923b23186ee90fcbf
Gerrit-Change-Number: 9459
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Feb 2018 06:28:34 +0000
Gerrit-HasComments: No