You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/07/10 19:15:30 UTC

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20181


Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
---
M be/src/util/codec.cc
M be/src/util/codec.h
2 files changed, 5 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/20181/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6: Code-Review+2

Thanks, this looks good to me


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 21:23:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9495/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 21:27:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13525/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 17:00:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py
File tests/custom_cluster/test_hive_sequence_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py@34
PS3, Line 34: 
> I'll look into a little refactoring.
Done


http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py@58
PS3, Line 58: 
> Nit: Since this test function is so similar to the test_hive_text_codec_int
Done


http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py
File tests/custom_cluster/test_hive_text_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py@30
PS3, Line 30: 
> Yes. It looks like text had specifically patched for this in https://github
hdfs-text-scanner.cc code is for a slightly different case (THdfsCompression::DEFLATE) where-as DefaultCodec/DeflateCodec map to THdfsCompression::DEFAULT.



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 21:30:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Removed -Verified by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20181

to look at the new patch set (#3).

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

Adds tests for sequence files. Excludes reading zstd due to
IMPALA-12276. Updates text file test as IMPALA-8721 is fixed.

Testing:
- manual exhaustive run of updated tests

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
---
M be/src/util/codec.cc
M be/src/util/codec.h
A tests/custom_cluster/test_hive_sequence_codec_interop.py
M tests/custom_cluster/test_hive_text_codec_interop.py
4 files changed, 121 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/20181/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13530/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 18:58:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9498/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:43:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13507/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 19:39:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9495/


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:39:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20181/1/be/src/util/codec.cc
File be/src/util/codec.cc:

http://gerrit.cloudera.org:8080/#/c/20181/1/be/src/util/codec.cc@59
PS1, Line 59:     {DEFLATE_COMPRESSION, THdfsCompression::DEFAULT},
I should be able to create a test case for this, just haven't figured it out yet.



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 19:18:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13509/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 10 Jul 2023 23:33:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20181

to look at the new patch set (#2).

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

Adds tests for sequence files. Excludes reading zstd due to
IMPALA-12276. Updates text file test as IMPALA-8721 is fixed.

Testing:
- manual exhaustive run of updated tests

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
---
M be/src/util/codec.cc
M be/src/util/codec.h
A tests/custom_cluster/test_hive_sequence_codec_interop.py
M tests/custom_cluster/test_hive_text_codec_interop.py
4 files changed, 121 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/20181/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6: Verified+1

Failed on a different transient (test_delete_partitioned). Overriding.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 14:49:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20181

to look at the new patch set (#4).

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

DeflateCodec isn't an issue with text files because they don't include a
codec header. Sequence files do, which we check on decompress.

Moves TestTextInterop to a E2E test since it doesn't require any special
startup options and refactors out test running to be format-agnostic.
Updates text file test as IMPALA-8721 is fixed. Adds tests for sequence
files; excludes reading zstd due to IMPALA-12276.

Testing:
- manual exhaustive run of updated tests

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
---
M be/src/util/codec.cc
M be/src/util/codec.h
R tests/query_test/test_hive_codec_interop.py
3 files changed, 49 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/20181/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20181/1/be/src/util/codec.cc
File be/src/util/codec.cc:

http://gerrit.cloudera.org:8080/#/c/20181/1/be/src/util/codec.cc@59
PS1, Line 59:     {DEFLATE_COMPRESSION, THdfsCompression::DEFAULT},
> I should be able to create a test case for this, just haven't figured it ou
Done


http://gerrit.cloudera.org:8080/#/c/20181/2/tests/custom_cluster/test_hive_sequence_codec_interop.py
File tests/custom_cluster/test_hive_sequence_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/2/tests/custom_cluster/test_hive_sequence_codec_interop.py@18
PS2, Line 18: text
> nit. a nit? maybe sequence
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 16:41:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 4: Code-Review+1

(3 comments)

LGTM

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py
File tests/query_test/test_hive_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@68
PS4, Line 68: 
nit. unnecessary newline?


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@109
PS4, Line 109: 
nit. unnecessary newline?


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@123
PS4, Line 123: 
nit. unnecessary newline?



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 15:01:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 5:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py
File tests/query_test/test_hive_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@a61
PS4, Line 61:     source_table = "{0}.{1}".format(unique_database, "t1_source")
This tested that Hive could read a textfile written by Impala. But it didn't test any particular compression, so that didn't seem very useful.


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@50
PS4, Line 50: ImpalaTestSuite
> TestFileCodecInterop?
That makes more sense. I misunderstood the super call, and calling super(TestFileCodecInterop, cls).add_test_dimensions fails because it's not defined on ImpalaTestSuite. Not sure why this one works.


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@68
PS4, Line 68: 
> nit. unnecessary newline?
I liked the formatting better, but I can remove it.


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@109
PS4, Line 109: 
> nit. unnecessary newline?
Ack


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@112
PS4, Line 112: TestFileCodecInterop
> TestTextInterop
Ah, I misunderstood this call.


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@123
PS4, Line 123: 
> nit. unnecessary newline?
Ack


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@126
PS4, Line 126: TestFileCodecInterop
> TestSequenceInterop
Ack


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@129
PS4, Line 129:   @pytest.mark.execute_serially
> I think we can remove this
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 18:05:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 4:

(5 comments)

Thanks for the refactoring, just a couple comments around how super() is getting called here.

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py
File tests/query_test/test_hive_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@50
PS4, Line 50: ImpalaTestSuite
TestFileCodecInterop?

The original test seems to have been doing something subtle here (calling grandparent BaseTestSuite::add_test_dimensions() rather than parent ImpalaTestSuite::add_test_dimensions()), and it ends up without all the default dimensions. Another way to do that is to directly initialize ImpalaTestMatrix. Here is an example from query_test/test_decimal_casting.py:

  @classmethod
  def add_test_dimensions(cls):
    cls.ImpalaTestMatrix = ImpalaTestMatrix()
    cls.ImpalaTestMatrix.add_dimension(ImpalaTestDimension('decimal_type',
        *TestDecimalCasting.DECIMAL_TYPES_MAP[cls.exploration_strategy()]))
    cls.ImpalaTestMatrix.add_dimension(
        ImpalaTestDimension('cast_from', *TestDecimalCasting.CAST_FROM))
    cls.ImpalaTestMatrix.add_dimension(create_exec_option_dimension_from_dict(
        {'decimal_v2': ['false','true']}))

So, this function could just be:
cls.ImpalaTestMatrix = ImpalaTestMatrix()
Then the subclasses can add what dimensions they want (or no dimensions as the case may be).

If we do that, I think the add_dimension()/add_constraint() calls in this method are unnecessary.


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@71
PS4, Line 71:     for codec in codecs:
Nit: (Not required for approval, just a bonus that comes from this not being a custom cluster test): When this was a custom cluster test, it made sense to test all the codecs in a single test, because it avoids Impala cluster restarts.

Now that we aren't doing Impala restarts, a different option is to make the codec a test dimension. That would allow all the codecs to run in parallel.

An example is test_failpoints.py:
https://github.com/apache/impala/blob/master/tests/failure/test_failpoints.py#L78-L79
https://github.com/apache/impala/blob/master/tests/failure/test_failpoints.py#L38-L39


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@112
PS4, Line 112: TestFileCodecInterop
TestTextInterop

(We're currently calling on the grandparent, not the parent class)


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@126
PS4, Line 126: TestFileCodecInterop
TestSequenceInterop


http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@129
PS4, Line 129:   @pytest.mark.execute_serially
I think we can remove this



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 15:38:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

LGTM

http://gerrit.cloudera.org:8080/#/c/20181/2/tests/custom_cluster/test_hive_sequence_codec_interop.py
File tests/custom_cluster/test_hive_sequence_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/2/tests/custom_cluster/test_hive_sequence_codec_interop.py@18
PS2, Line 18: text
nit. a nit? maybe sequence



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 15:17:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 3:

(3 comments)

Thanks for putting together this fix, I'm fine with this as-is, but a couple questions and nits.

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py
File tests/custom_cluster/test_hive_sequence_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py@34
PS3, Line 34: CustomClusterTestSuite
Not your change, but I'm not sure why these are custom cluster tests rather than end-to-end tests. It doesn't seem like we would need to restart Impala to test this.


http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py@58
PS3, Line 58: test_hive_impala_interop
Nit: Since this test function is so similar to the test_hive_text_codec_interop.py one, it might make sense to have a shared base class that provides a function used by both.


http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py
File tests/custom_cluster/test_hive_text_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py@30
PS3, Line 30: 'deflate'
Was deflate already working for text? i.e. the bug was specific to sequence files?



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 18:11:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py
File tests/custom_cluster/test_hive_sequence_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_sequence_codec_interop.py@34
PS3, Line 34: CustomClusterTestSuite
> Not your change, but I'm not sure why these are custom cluster tests rather
I'll look into a little refactoring.


http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py
File tests/custom_cluster/test_hive_text_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/3/tests/custom_cluster/test_hive_text_codec_interop.py@30
PS3, Line 30: 'deflate'
> Was deflate already working for text? i.e. the bug was specific to sequence
Yes. It looks like text had specifically patched for this in https://github.com/apache/impala/blob/master/be/src/exec/text/hdfs-text-scanner.cc#L235-L237. I'm going to look at removing the text-specific patch in favor of this general patch, and I'll add some more explanation to the patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 18:26:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py
File tests/query_test/test_hive_codec_interop.py:

http://gerrit.cloudera.org:8080/#/c/20181/4/tests/query_test/test_hive_codec_interop.py@71
PS4, Line 71:     self.run_stmt_in_hiv
> Nit: (Not required for approval, just a bonus that comes from this not bein
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Jul 2023 18:36:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9498/


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 05:56:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13527/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Jul 2023 21:49:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Patch Set 6:

Ran into https://issues.apache.org/jira/browse/IMPALA-12187, re-running.


-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 13 Jul 2023 01:43:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................


Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20181

to look at the new patch set (#6).

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

DeflateCodec isn't an issue with text files because they don't include a
codec header. Sequence files do, which we check on decompress.

Moves TestTextInterop to a E2E test since it doesn't require any special
startup options and refactors out test running to be format-agnostic.
Updates text file test as IMPALA-8721 is fixed. Removes creating a table
in Impala for Hive to read, as it didn't test anything new. Adds tests
for sequence files; excludes reading zstd due to IMPALA-12276.

Testing:
- manual exhaustive run of updated tests

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
---
M be/src/util/codec.cc
M be/src/util/codec.h
D tests/custom_cluster/test_hive_text_codec_interop.py
A tests/query_test/test_hive_codec_interop.py
4 files changed, 126 insertions(+), 115 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/20181/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12275: Read files written with DeflateCodec

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20181 )

Change subject: IMPALA-12275: Read files written with DeflateCodec
......................................................................

IMPALA-12275: Read files written with DeflateCodec

DeflateCodec is an alias to DefaultCodec. Impala works with
DefaultCodec. Fixes reading files written with DeflateCodec.

DeflateCodec isn't an issue with text files because they don't include a
codec header. Sequence files do, which we check on decompress.

Moves TestTextInterop to a E2E test since it doesn't require any special
startup options and refactors out test running to be format-agnostic.
Updates text file test as IMPALA-8721 is fixed. Removes creating a table
in Impala for Hive to read, as it didn't test anything new. Adds tests
for sequence files; excludes reading zstd due to IMPALA-12276.

Testing:
- manual exhaustive run of updated tests

Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Reviewed-on: http://gerrit.cloudera.org:8080/20181
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M be/src/util/codec.cc
M be/src/util/codec.h
D tests/custom_cluster/test_hive_text_codec_interop.py
A tests/query_test/test_hive_codec_interop.py
4 files changed, 126 insertions(+), 115 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Michael Smith: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20181
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id5ec1d0345ae35597f6aade9d8b9eef2257efeba
Gerrit-Change-Number: 20181
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>