You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Michael Ho (Code Review)" <ge...@cloudera.org> on 2016/07/05 21:30:54 UTC

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Michael Ho has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/3575

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................

IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

A new test has been added to test the decompression of a 2GB
compressed text file.

Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
---
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/collection-value-builder.h
M be/src/runtime/free-pool-test.cc
M be/src/runtime/free-pool.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/string-buffer-test.cc
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/bit-util.h
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M common/thrift/generate_error_codes.py
M testdata/bin/create-load-data.sh
A testdata/compressed_formats/compressed_payload.snap
M tests/query_test/test_compressed_formats.py
29 files changed, 366 insertions(+), 340 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/75/3575/1
-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 1:

This is mostly identical to https://gerrit.cloudera.org/#/c/2781/ which was backed out. The major changes made were related to changes in test_compressed_formats.py. Specifically, we will no longer install python_snappy as there is cyclic dependency with the toolchain (i.e. bootstrap_virtualenv.py relies on the existence of toolchain but that's not necessarily true with a clean tree). This change adds a snappy compressed payload into testdata which will be used as the building block for the test file.

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 2: Code-Review+1

Carry Tim's +1 forward.

Exhaustive tests: http://sandbox.jenkins.cloudera.com/job/impala-private-build-and-test/3598/

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 3: Code-Review+2

Carry +2 forward. Will GVM now.

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 1:

Exhaustive build: http://sandbox.jenkins.cloudera.com/job/impala-private-build-and-test/3585/

A new run is in progress after rebase.

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/3575/1/testdata/compressed_formats/compressed_payload.snap
File testdata/compressed_formats/compressed_payload.snap:

> Can you add a README that describes the file?
Done


http://gerrit.cloudera.org:8080/#/c/3575/1/tests/query_test/test_compressed_formats.py
File tests/query_test/test_compressed_formats.py:

PS1, Line 190: hdfs_cat
> I think you may want to call .wait() ion this. I'm not sure if the resource
Done


Line 193:     for line in hdfs_cat.stdout:
> Why not 
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 3: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 2: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 2:

Can we merge this? I'm waiting to rebase a patch on top of it.

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/3575

to look at the new patch set (#2).

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................

IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

A new test has been added to test the decompression of a 2GB
snappy block compressed text file.

Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
---
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/collection-value-builder.h
M be/src/runtime/free-pool-test.cc
M be/src/runtime/free-pool.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/string-buffer-test.cc
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/bit-util.h
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M common/thrift/generate_error_codes.py
M testdata/bin/create-load-data.sh
A testdata/compressed_formats/README
A testdata/compressed_formats/compressed_payload.snap
M tests/query_test/test_compressed_formats.py
30 files changed, 370 insertions(+), 340 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/75/3575/2
-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


IMPALA-1619: Support 64-bit allocations.

This change extends MemPool, FreePool and StringBuffer to support
64-bit allocations, fixes a bug in decompressor and extends various
places in the code to support 64-bit allocation sizes. With this
change, the text scanner can now decompress compressed files larger
than 1GB.

Note that the UDF interfaces FunctionContext::Allocate() and
FunctionContext::Reallocate() still use 32-bit for the input
argument to avoid breaking compatibility.

In addition, the byte size of a tuple is still assumed to be
within 32-bit. If it needs to be upgraded to 64-bit, it will be
done in a separate change.

A new test has been added to test the decompression of a 2GB
snappy block compressed text file.

Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Reviewed-on: http://gerrit.cloudera.org:8080/3575
Reviewed-by: Michael Ho <kw...@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/exec/delimited-text-parser.cc
M be/src/exec/delimited-text-parser.h
M be/src/exec/delimited-text-parser.inline.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/collection-value-builder.h
M be/src/runtime/free-pool-test.cc
M be/src/runtime/free-pool.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-pool.h
M be/src/runtime/string-buffer-test.cc
M be/src/runtime/string-buffer.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/bit-util.h
M be/src/util/codec.cc
M be/src/util/codec.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M common/thrift/generate_error_codes.py
M testdata/bin/create-load-data.sh
A testdata/compressed_formats/README
A testdata/compressed_formats/compressed_payload.snap
M tests/query_test/test_compressed_formats.py
30 files changed, 370 insertions(+), 340 deletions(-)

Approvals:
  Michael Ho: Looks good to me, approved
  Internal Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 1: Code-Review+1

(3 comments)

http://gerrit.cloudera.org:8080/#/c/3575/1/testdata/compressed_formats/compressed_payload.snap
File testdata/compressed_formats/compressed_payload.snap:

Can you add a README that describes the file?


http://gerrit.cloudera.org:8080/#/c/3575/1/tests/query_test/test_compressed_formats.py
File tests/query_test/test_compressed_formats.py:

PS1, Line 190: hdfs_cat
I think you may want to call .wait() ion this. I'm not sure if the resources are released otherwise.


Line 193:     for line in hdfs_cat.stdout:
Why not 

  compressed_payload = hdfs_cat.stdout.read()


-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-CR](cdh5-trunk) IMPALA-1619: Support 64-bit allocations.

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-1619: Support 64-bit allocations.
......................................................................


Patch Set 1:

Packaging build passed: http://golden.jenkins.cloudera.com/job/generic-package-sles64-11-impala/

-- 
To view, visit http://gerrit.cloudera.org:8080/3575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic1af1564953ac02aca2728646973199381c86e5f
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-HasComments: No