You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2020/01/21 02:31:57 UTC
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15073
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
IMPALA-6772: Add ORC 1.6.2 to toolchain
Add ORC 1.6.2 including 6 fixes. The patch files are modified to exclude
changes on binary files in tests. They will cause failures in patching.
Run test_scanners.py for orc/def/block locally.
Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
---
M buildall.sh
A source/orc/orc-1.6.2-patches/0001-ORC-580-C-Verify-ColumnEncodings-in-StripeFooter-463.patch
A source/orc/orc-1.6.2-patches/0002-ORC-581-C-Verify-fieldNames-size-for-STRUCT-types.patch
A source/orc/orc-1.6.2-patches/0003-ORC-586-C-fix-memory-leak-in-StructColumnReader.patch
A source/orc/orc-1.6.2-patches/0004-ORC-589-C-add-checks-about-negative-dictionary-entry.patch
A source/orc/orc-1.6.2-patches/0005-ORC-590-C-added-check-for-missing-LENGTH-stream-in-S.patch
A source/orc/orc-1.6.2-patches/0006-ORC-591-C-Check-missing-blob-stream-for-StringDictio.patch
7 files changed, 723 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/73/15073/1
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15073/2/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/15073/2/buildall.sh@352
PS2, Line 352: ORC_VERSION=1.5.5-p1 $SOURCE_DIR/source/orc/build.sh
> Do we need to build the old versions? Other buildall.sh-s put the build to
Done
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 13:13:09 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 14:51:46 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
IMPALA-6772: Add ORC 1.6.2 to toolchain
Add ORC 1.6.2 including 6 fixes. The patch files are modified to exclude
changes on binary files in tests. They will cause failures in patching.
Tests:
- Run query_test for orc/def/block locally.
- Builds succeed in all supported platforms.
Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Reviewed-on: http://gerrit.cloudera.org:8080/15073
Tested-by: Quanlong Huang <hu...@gmail.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
---
M buildall.sh
A source/orc/orc-1.6.2-patches/0001-ORC-580-C-Verify-ColumnEncodings-in-StripeFooter-463.patch
A source/orc/orc-1.6.2-patches/0002-ORC-581-C-Verify-fieldNames-size-for-STRUCT-types.patch
A source/orc/orc-1.6.2-patches/0003-ORC-586-C-fix-memory-leak-in-StructColumnReader.patch
A source/orc/orc-1.6.2-patches/0004-ORC-589-C-add-checks-about-negative-dictionary-entry.patch
A source/orc/orc-1.6.2-patches/0005-ORC-590-C-added-check-for-missing-LENGTH-stream-in-S.patch
A source/orc/orc-1.6.2-patches/0006-ORC-591-C-Check-missing-blob-stream-for-StringDictio.patch
7 files changed, 727 insertions(+), 2 deletions(-)
Approvals:
Quanlong Huang: Verified
Csaba Ringhofer: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Norbert Luksa (Code Review)" <ge...@cloudera.org>.
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh@347
PS3, Line 347: export SNAPPY_VERSION=1.1.4
Shouldn't these versions get updated to match the ones in the ORC lib?
https://github.com/apache/orc/blob/master/cmake_modules/ThirdpartyToolchain.cmake#L14
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 14:29:06 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 14:22:53 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15073
to look at the new patch set (#2).
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
IMPALA-6772: Add ORC 1.6.2 to toolchain
Add ORC 1.6.2 including 6 fixes. The patch files are modified to exclude
changes on binary files in tests. They will cause failures in patching.
Tests:
- Run query_test for orc/def/block locally.
- Builds succeed in all supported platforms.
Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
---
M buildall.sh
A source/orc/orc-1.6.2-patches/0001-ORC-580-C-Verify-ColumnEncodings-in-StripeFooter-463.patch
A source/orc/orc-1.6.2-patches/0002-ORC-581-C-Verify-fieldNames-size-for-STRUCT-types.patch
A source/orc/orc-1.6.2-patches/0003-ORC-586-C-fix-memory-leak-in-StructColumnReader.patch
A source/orc/orc-1.6.2-patches/0004-ORC-589-C-add-checks-about-negative-dictionary-entry.patch
A source/orc/orc-1.6.2-patches/0005-ORC-590-C-added-check-for-missing-LENGTH-stream-in-S.patch
A source/orc/orc-1.6.2-patches/0006-ORC-591-C-Check-missing-blob-stream-for-StringDictio.patch
7 files changed, 723 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/73/15073/2
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Norbert Luksa (Code Review)" <ge...@cloudera.org>.
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh@347
PS3, Line 347: export SNAPPY_VERSION=1.1.4
> IIUC, we use these versions in our native-toolchain to have consistent beha
I see, thanks for clarifying!
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 14:53:06 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15073/2/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/15073/2/buildall.sh@352
PS2, Line 352: ORC_VERSION=1.6.2-p6 $SOURCE_DIR/source/orc/build.sh
Do we need to build the old versions? Other buildall.sh-s put the build to if (( BUILD_HISTORICAL )); then block
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 10:13:40 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/15073/3/buildall.sh@347
PS3, Line 347: export SNAPPY_VERSION=1.1.4
> Shouldn't these versions get updated to match the ones in the ORC lib?
IIUC, we use these versions in our native-toolchain to have consistent behaviors across different libs.
ORC library supports customized third party libraries by cmake variables like -DSNAPPY_HOME=... They're specified here: https://github.com/cloudera/native-toolchain/blob/master/source/orc/build.sh#L53
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 14:47:43 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15073 )
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Jan 2020 10:59:41 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-6772: Add ORC 1.6.2 to toolchain
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Tim Armstrong, Csaba Ringhofer,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15073
to look at the new patch set (#3).
Change subject: IMPALA-6772: Add ORC 1.6.2 to toolchain
......................................................................
IMPALA-6772: Add ORC 1.6.2 to toolchain
Add ORC 1.6.2 including 6 fixes. The patch files are modified to exclude
changes on binary files in tests. They will cause failures in patching.
Tests:
- Run query_test for orc/def/block locally.
- Builds succeed in all supported platforms.
Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
---
M buildall.sh
A source/orc/orc-1.6.2-patches/0001-ORC-580-C-Verify-ColumnEncodings-in-StripeFooter-463.patch
A source/orc/orc-1.6.2-patches/0002-ORC-581-C-Verify-fieldNames-size-for-STRUCT-types.patch
A source/orc/orc-1.6.2-patches/0003-ORC-586-C-fix-memory-leak-in-StructColumnReader.patch
A source/orc/orc-1.6.2-patches/0004-ORC-589-C-add-checks-about-negative-dictionary-entry.patch
A source/orc/orc-1.6.2-patches/0005-ORC-590-C-added-check-for-missing-LENGTH-stream-in-S.patch
A source/orc/orc-1.6.2-patches/0006-ORC-591-C-Check-missing-blob-stream-for-StringDictio.patch
7 files changed, 727 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/73/15073/3
--
To view, visit http://gerrit.cloudera.org:8080/15073
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I62734f791fd8fc098579f488a869a2153ebad919
Gerrit-Change-Number: 15073
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>