You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2018/03/08 17:36:33 UTC

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9556


Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................

IMPALA-6589: remove invalid DCHECK in parquet reader

The DCHECK was only valid if the Parquet file metadata is internally
consistent, with the number of values reported by the metadata
matching the number of encoded levels.

The DCHECK was intended to directly detect misuse of the RleBatchDecoder
interface, which would lead to incorrect results. However, our other
test coverage for reading Parquet files is sufficient to test the
correctness of level decoding.

Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
---
M be/src/exec/parquet-column-readers.cc
1 file changed, 0 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/9556/1
-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
It is feasible to add a test file that would have hit this and now is correctly handled elsewhere?



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Comment-Date: Thu, 08 Mar 2018 17:43:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................

IMPALA-6589: remove invalid DCHECK in parquet reader

The DCHECK was only valid if the Parquet file metadata is internally
consistent, with the number of values reported by the metadata
matching the number of encoded levels.

The DCHECK was intended to directly detect misuse of the RleBatchDecoder
interface, which would lead to incorrect results. However, our other
test coverage for reading Parquet files is sufficient to test the
correctness of level decoding.

Testing:
Added a minimal corrupt test file that reproduces the issue.

Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Reviewed-on: http://gerrit.cloudera.org:8080/9556
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/exec/parquet-column-readers.cc
M be/src/exec/parquet-column-readers.h
M testdata/data/README
A testdata/data/num_values_def_levels_mismatch.parquet
A testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
M tests/query_test/test_scanners.py
6 files changed, 50 insertions(+), 4 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Pranay Singh, Lars Volker, Csaba Ringhofer, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9556

to look at the new patch set (#4).

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................

IMPALA-6589: remove invalid DCHECK in parquet reader

The DCHECK was only valid if the Parquet file metadata is internally
consistent, with the number of values reported by the metadata
matching the number of encoded levels.

The DCHECK was intended to directly detect misuse of the RleBatchDecoder
interface, which would lead to incorrect results. However, our other
test coverage for reading Parquet files is sufficient to test the
correctness of level decoding.

Testing:
Added a minimal corrupt test file that reproduces the issue.

Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
---
M be/src/exec/parquet-column-readers.cc
M be/src/exec/parquet-column-readers.h
M testdata/data/README
A testdata/data/num_values_def_levels_mismatch.parquet
A testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
M tests/query_test/test_scanners.py
6 files changed, 50 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/9556/4
-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/9556/2/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/2/be/src/exec/parquet-column-readers.cc@1207
PS2, Line 1207:     rep_level_ = rep_levels_.ReadLevel();
Can you also add a similar check for the repetition level?


http://gerrit.cloudera.org:8080/#/c/9556/2/testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
File testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test:

http://gerrit.cloudera.org:8080/#/c/9556/2/testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test@7
PS2, Line 7: 
nit: unnecessary line


http://gerrit.cloudera.org:8080/#/c/9556/2/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:

http://gerrit.cloudera.org:8080/#/c/9556/2/tests/query_test/test_scanners.py@433
PS2, Line 433: 
nit: unnecessary line



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 10 Mar 2018 12:37:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 3: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Mar 2018 09:36:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/9556/2/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/2/be/src/exec/parquet-column-readers.cc@1207
PS2, Line 1207:     rep_level_ = rep_levels_.ReadLevel();
> Can you also add a similar check for the repetition level?
Done


http://gerrit.cloudera.org:8080/#/c/9556/2/testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
File testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test:

http://gerrit.cloudera.org:8080/#/c/9556/2/testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test@7
PS2, Line 7: 
> nit: unnecessary line
Done


http://gerrit.cloudera.org:8080/#/c/9556/2/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:

http://gerrit.cloudera.org:8080/#/c/9556/2/tests/query_test/test_scanners.py@433
PS2, Line 433: 
> nit: unnecessary line
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Mar 2018 00:00:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> Nit: good to check that num_cached_levels GE 0
Done


http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> My guess is that this can occur if a corrupt Parquet file contains a litera
Csaba does raise an interesting point - the Parquet RLE spec may allow 0-length literal runs, which our code might treat as meaning "end of input". I guess that the spec didn't intend to allow this.

I could trigger this DCHECK without exercising that particular edge case, but it seems like it might be worth pursuing that line of thought further.


http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> It is feasible to add a test file that would have hit this and now is corre
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Mar 2018 22:30:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> Nit: good to check that num_cached_levels GE 0
My guess is that this can occur if a corrupt Parquet file contains a literal count with value 0 (coded as 0x01) and FillCache was last called via ParquetLevelDecoder::ReadLevel(). FillCache should return false in this case, and ReadLevel()'s caller, BaseScalarColumnReader::NextLevels() should check for  HdfsParquetScanner::INVALID_LEVEL. This way a proper warning/error could be returned instead of crashing.



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Comment-Date: Thu, 08 Mar 2018 20:51:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Pranay Singh, Lars Volker, Csaba Ringhofer, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9556

to look at the new patch set (#3).

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................

IMPALA-6589: remove invalid DCHECK in parquet reader

The DCHECK was only valid if the Parquet file metadata is internally
consistent, with the number of values reported by the metadata
matching the number of encoded levels.

The DCHECK was intended to directly detect misuse of the RleBatchDecoder
interface, which would lead to incorrect results. However, our other
test coverage for reading Parquet files is sufficient to test the
correctness of level decoding.

Testing:
Added a minimal corrupt test file that reproduces the issue.

Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
---
M be/src/exec/parquet-column-readers.cc
M be/src/exec/parquet-column-readers.h
M testdata/data/README
A testdata/data/num_values_def_levels_mismatch.parquet
A testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
M tests/query_test/test_scanners.py
6 files changed, 50 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/9556/3
-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2124/


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 23:02:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 23:00:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Pranay Singh, Lars Volker, Csaba Ringhofer, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9556

to look at the new patch set (#2).

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................

IMPALA-6589: remove invalid DCHECK in parquet reader

The DCHECK was only valid if the Parquet file metadata is internally
consistent, with the number of values reported by the metadata
matching the number of encoded levels.

The DCHECK was intended to directly detect misuse of the RleBatchDecoder
interface, which would lead to incorrect results. However, our other
test coverage for reading Parquet files is sufficient to test the
correctness of level decoding.

Testing:
Added a minimal corrupt test file that reproduces the issue.

Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
---
M be/src/exec/parquet-column-readers.cc
M testdata/data/README
A testdata/data/num_values_def_levels_mismatch.parquet
A testdata/workloads/functional-query/queries/QueryTest/parquet-num-values-def-levels-mismatch.test
M tests/query_test/test_scanners.py
5 files changed, 36 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/9556/2
-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 17 Mar 2018 02:52:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 4: Code-Review+1

Rebase and fix a long line. Carry +1


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Mar 2018 00:46:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> It is feasible to add a test file that would have hit this and now is corre
Nit: good to check that num_cached_levels GE 0



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Comment-Date: Thu, 08 Mar 2018 17:46:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:

http://gerrit.cloudera.org:8080/#/c/9556/1/be/src/exec/parquet-column-readers.cc@a175
PS1, Line 175: 
> My guess is that this can occur if a corrupt Parquet file contains a litera
It looks like BaseScalarColumnReader::NextLevels() implicitly checks for INVALID_LEVEL with the < 0 check btw. I did restructure that logic so it produced a more sensible error.



-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Mar 2018 22:32:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6589: remove invalid DCHECK in parquet reader

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9556 )

Change subject: IMPALA-6589: remove invalid DCHECK in parquet reader
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idd6e09f8c8cca8991be5b5b379f6420adaa97daa
Gerrit-Change-Number: 9556
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 16 Mar 2018 23:02:30 +0000
Gerrit-HasComments: No