You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2024/06/30 13:28:27 UTC
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21559
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
IMPALA-13161: Fix column index overflow in DelimitedTextParser
DelimitedTextParser tracks the current column index inside the current
row that is under parsing. The row could have arbitrary numbers of
fields. The index, 'column_idx_', is defined as int type which could
overflow when there are more than 2^31 fields in the row. This index is
only used to check whether the current column should be materialized. It
doesn't make sense to track the index if it's larger than the number of
columns of the table.
This patch fixes the overflow issue by only bumping 'column_idx_' when
it's smaller than the number of columns of the table.
Tests
- Add e2e test
Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
---
M be/src/exec/delimited-text-parser.inline.h
M tests/query_test/test_delimited_text.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/21559/1
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16450/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 01 Jul 2024 03:19:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h
File be/src/exec/delimited-text-parser.inline.h:
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h@76
PS2, Line 76: if (column_idx_ < num_cols_) ++column_idx_;
> That file would be ambiguous if it had more than 1 column? How does the beh
The file won't be ambiguous. How to read and materialize the data is defined by the table schema (i.e. column types) and the tuple descriptor (i.e. what the query wants, mainly a bool array of 'is_materialized_col_').
How many columns in a text line is determined by the field delimiters. If one line has N field delimiters, then N+1 columns can be read out from that line. However, it depends on the table schema and tuple descriptor to select the columns. As this comment mentions, we use a two pass parsing approach:
https://github.com/apache/impala/blob/e2e45401e2bead4090fd5c562709db521cbc6d38/be/src/exec/hdfs-scanner.h#L60-L77
The first pass fills out the FieldLocation structs which contain where all the fields start and their lengths in the byte buffer.
The second pass uses the FieldLocation to materialize slots in the tuples.
AddColumn() is used in the first pass. If a text line has more columns than the table schema, they won't be added into the FieldLocations. This is judged by ReturnCurrentColumn():
bool ReturnCurrentColumn() const {
return column_idx_ < num_cols_ && is_materialized_col_[column_idx_];
}
'column_idx_' is the index of the column in the text line. It starts from the number of partitions (not starts from 0). It's only used to compare with num_cols_ and num_partition_keys_ so we don't need to bump it if it's already larger than num_cols_.
The original code keeps bumping it even if it's overflowed. Then column_idx_ < num_cols_ passes and accessing is_materialized_col_[column_idx_] crashes. With this fix, column_idx_ won't be larger than num_cols_ so won't cause the crash. This is the only behavior change and it's not aware by the user or any caller of this function.
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 12 Jul 2024 06:27:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10771/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 30 Jun 2024 13:36:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
IMPALA-13161: Fix column index overflow in DelimitedTextParser
DelimitedTextParser tracks the current column index inside the current
row that is under parsing. The row could have arbitrary numbers of
fields. The index, 'column_idx_', is defined as int type which could
overflow when there are more than 2^31 fields in the row. This index is
only used to check whether the current column should be materialized. It
doesn't make sense to track the index if it's larger than the number of
columns of the table.
This patch fixes the overflow issue by only bumping 'column_idx_' when
it's smaller than the number of columns of the table.
Tests
- Add e2e test
Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Reviewed-on: http://gerrit.cloudera.org:8080/21559
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/delimited-text-parser.inline.h
M tests/query_test/test_delimited_text.py
2 files changed, 19 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2: Code-Review+1
LGTM
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Jul 2024 22:47:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 17 Jul 2024 04:01:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16449/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 30 Jun 2024 13:53:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 30 Jun 2024 18:42:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h
File be/src/exec/delimited-text-parser.inline.h:
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h@76
PS2, Line 76: if (column_idx_ < num_cols_) ++column_idx_;
Why would we call "AddColumn" more times than num_cols_?
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 09 Jul 2024 00:27:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Jul 2024 22:53:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h
File be/src/exec/delimited-text-parser.inline.h:
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h@76
PS2, Line 76: if (column_idx_ < num_cols_) ++column_idx_;
> num_cols_ comes from the table metadata. The actual data line could have mo
That file would be ambiguous if it had more than 1 column? How does the behavior change for a file with 2 columns and many field delimiters (but not enough to overflow)?
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 10 Jul 2024 17:55:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h
File be/src/exec/delimited-text-parser.inline.h:
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h@76
PS2, Line 76: if (column_idx_ < num_cols_) ++column_idx_;
> The file won't be ambiguous. How to read and materialize the data is define
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 12 Jul 2024 15:24:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Jul 2024 22:54:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h
File be/src/exec/delimited-text-parser.inline.h:
http://gerrit.cloudera.org:8080/#/c/21559/2/be/src/exec/delimited-text-parser.inline.h@76
PS2, Line 76: if (column_idx_ < num_cols_) ++column_idx_;
> Why would we call "AddColumn" more times than num_cols_?
num_cols_ comes from the table metadata. The actual data line could have more field delimiters so we need to iterate them until the next line delimiter.
https://github.com/apache/impala/blob/e419545c1c2b6fa0b6d23643d3b2eb0faa4dce52/be/src/exec/delimited-text-parser.h#L110-L112
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 10 Jul 2024 04:41:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10798/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 16 Jul 2024 22:54:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21559
to look at the new patch set (#2).
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
IMPALA-13161: Fix column index overflow in DelimitedTextParser
DelimitedTextParser tracks the current column index inside the current
row that is under parsing. The row could have arbitrary numbers of
fields. The index, 'column_idx_', is defined as int type which could
overflow when there are more than 2^31 fields in the row. This index is
only used to check whether the current column should be materialized. It
doesn't make sense to track the index if it's larger than the number of
columns of the table.
This patch fixes the overflow issue by only bumping 'column_idx_' when
it's smaller than the number of columns of the table.
Tests
- Add e2e test
Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
---
M be/src/exec/delimited-text-parser.inline.h
M tests/query_test/test_delimited_text.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/21559/2
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/1/tests/query_test/test_delimited_text.py
File tests/query_test/test_delimited_text.py:
http://gerrit.cloudera.org:8080/#/c/21559/1/tests/query_test/test_delimited_text.py@90
PS1, Line 90:
> flake8: E703 statement ends with a semicolon
Done
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 01 Jul 2024 02:54:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13161: Fix column index overflow in DelimitedTextParser
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21559 )
Change subject: IMPALA-13161: Fix column index overflow in DelimitedTextParser
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21559/1/tests/query_test/test_delimited_text.py
File tests/query_test/test_delimited_text.py:
http://gerrit.cloudera.org:8080/#/c/21559/1/tests/query_test/test_delimited_text.py@90
PS1, Line 90: ;
flake8: E703 statement ends with a semicolon
--
To view, visit http://gerrit.cloudera.org:8080/21559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I527a8971e92e270d5576c2155e4622dd6d43d745
Gerrit-Change-Number: 21559
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 30 Jun 2024 13:29:36 +0000
Gerrit-HasComments: Yes