You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zinway (Code Review)" <ge...@cloudera.org> on 2023/12/27 07:34:25 UTC

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Zinway has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20834


Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue being addressed occurs when scanning Parquet tables with a row size greater than 4096 bytes and a row batch size greater than 1024. A heap-buffer-overflow was detected by AddressSanitizer, indicating that there was a write operation beyond the allocated buffer space.

**Root Cause Analysis:**
The error log provided by AddressSanitizer points to a heap-buffer-overflow, where memory is being accessed beyond the allocated region. This is happening in the context of the `HdfsParquetScanner` and `ScratchTupleBatch` classes, specifically when handling large rows with a size greater than 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many columns and inserting data into it using Hive, then querying the table with Impala. The provided bash and Hive client scripts in IMPALA-12665 create a table and populate it with data to set up the conditions that trigger the bug.

**Technical Analysis:**
The `ScratchTupleBatch::Reset` method recalculates the `capacity` based on the size of the tuples and the fixed memory limit constants defined. When the row size exceeds 4096 bytes, the `capacity` is set to a value smaller than the default 1024. However, the `HdfsParquetScanner` method incorrectly assumes a `complete_micro_batch_` length of 1024, leading to an overflow when accessing memory beyond the newly adjusted capacity.

**Proposed Solution:**
The code change should ensure that the `complete_micro_batch_` length is updated to reflect the new `capacity` after the `ScratchTupleBatch::Reset` method is called. This would prevent accessing memory outside the allocated buffer and thus avoid the heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
2 files changed, 11 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 1
Gerrit-Owner: Zinway <zi...@gmail.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10149/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 10
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 21:31:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 8:

(7 comments)

Thank you for the detailed investigation!
The code looks good to me. I had a few minor comments, mainly about the test.

http://gerrit.cloudera.org:8080/#/c/20834/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20834/8//COMMIT_MSG@23
PS8, Line 23: inserting data using Hive, then querying with Impala.
            : Bash and Hive client scripts
From the test it seems like the error was also reproducible with Impala.


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@463
PS8, Line 463: class TestRowSizeGT4096Queries(ImpalaTestSuite):
nit: tests/query_test/test_scanners.py might be a better place for this class

Also, probably there is no need for this class, the test functions could be placed into TestParquet.


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@466
PS8, Line 466:     super(TestRowSizeGT4096Queries, cls).add_test_dimensions()
We should only execute this test when file_format == 'parquet' in the test vector. This will be resolved automatically if you move the test methods to test_scanners.py::TestParquet.


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@475
PS8, Line 475:   
nit: we use 4 spaces on continuation lines


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@477
PS8, Line 477:     # drop table
             :     drop_sql = "DROP TABLE IF EXISTS {0}".format(table_name)
             :     self.execute_query_expect_success(self.client, drop_sql)
We don't need this as this test runs in its own unique database.


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@484
PS8, Line 484:   
nit: we use 4 spaces on continuation lines


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@505
PS8, Line 505:   
nit: we use 4 spaces on continuation lines



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 8
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 21:32:49 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_queries.py
3 files changed, 63 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 6
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc@2367
PS2, Line 2367:     complete_micro_batch_.AdjustLength(scratch_batch_->capacity);
> complete_micro_batch_ is used here only and scratch_batch_->capacity in lat
Let's add a DCHECK to verify that.



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 7
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 18:34:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14930/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 8
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 04:23:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 9
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 07:26:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Reviewed-on: http://gerrit.cloudera.org:8080/20834
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_scanners.py
3 files changed, 51 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 11
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20834/1/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@2365
PS1, Line 2365:     // Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
line too long (105 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 1
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Dec 2023 07:35:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20834/1/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@2365
PS1, Line 2365:     // Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
> line too long (105 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 1
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Comment-Date: Wed, 27 Dec 2023 08:06:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 4:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py@480
PS4, Line 480: n
flake8: E501 line too long (107 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py@485
PS4, Line 485: E
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py@486
PS4, Line 486: L
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py@495
PS4, Line 495: (
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/20834/4/tests/query_test/test_queries.py@496
PS4, Line 496: a
flake8: E501 line too long (127 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 4
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 07:33:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue being addressed occurs when scanning Parquet tables with a row size greater than 4096 bytes and a row batch size greater than 1024. A heap-buffer-overflow was detected by AddressSanitizer, indicating that there was a write operation beyond the allocated buffer space.

**Root Cause Analysis:**
The error log provided by AddressSanitizer points to a heap-buffer-overflow, where memory is being accessed beyond the allocated region. This is happening in the context of the `HdfsParquetScanner` and `ScratchTupleBatch` classes, specifically when handling large rows with a size greater than 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many columns and inserting data into it using Hive, then querying the table with Impala. The provided bash and Hive client scripts in IMPALA-12665 create a table and populate it with data to set up the conditions that trigger the bug.

**Technical Analysis:**
The `ScratchTupleBatch::Reset` method recalculates the `capacity` based on the size of the tuples and the fixed memory limit constants defined. When the row size exceeds 4096 bytes, the `capacity` is set to a value smaller than the default 1024. However, the `HdfsParquetScanner` method incorrectly assumes a `complete_micro_batch_` length of 1024, leading to an overflow when accessing memory beyond the newly adjusted capacity.

**Proposed Solution:**
The code change should ensure that the `complete_micro_batch_` length is updated to reflect the new `capacity` after the `ScratchTupleBatch::Reset` method is called. This would prevent accessing memory outside the allocated buffer and thus avoid the heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
2 files changed, 12 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc@2367
PS2, Line 2367:     complete_micro_batch_.AdjustLength(scratch_batch_->capacity);
> Let's add a DCHECK to verify that.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 03:51:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 9: Code-Review+2

Thanks again for fixing this issue!


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 9
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 21:31:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14911/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 6
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 08:32:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 10: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 10
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 21:31:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 9:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/20834/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20834/8//COMMIT_MSG@23
PS8, Line 23: inserting data using Hive, then querying with Impala.
            : Bash and Hive client scripts
> From the test it seems like the error was also reproducible with Impala.
Just for preparing data, use the following query utilizes certain functions in Hive to swiftly generate 2000 rows of data:
SELECT * FROM (
    SELECT posexplode(SPLIT(REPEAT(' ', 2000), ' ')) AS (id, val) FROM (SELECT 1) t
) s
LIMIT 2000;


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@463
PS8, Line 463: 
> nit: tests/query_test/test_scanners.py might be a better place for this cla
Done


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@466
PS8, Line 466: 
> We should only execute this test when file_format == 'parquet' in the test 
Done


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@475
PS8, Line 475: 
> nit: we use 4 spaces on continuation lines
Done


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@477
PS8, Line 477: 
             : 
             : 
> We don't need this as this test runs in its own unique database.
Done


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@484
PS8, Line 484: 
> nit: we use 4 spaces on continuation lines
Done


http://gerrit.cloudera.org:8080/#/c/20834/8/tests/query_test/test_queries.py@505
PS8, Line 505: 
> nit: we use 4 spaces on continuation lines
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 9
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 07:01:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_scanners.py
3 files changed, 51 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/9
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 9
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14836/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Comment-Date: Wed, 27 Dec 2023 08:17:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 10
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Jan 2024 01:51:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 2:

(3 comments)

I added a test case query_test/test_queries.py::TestRowSizeGT4096Queries::test_row_size_gt_4096_queries

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
> Impala commit messages try to stick to line limits of 72 characters. Not a 
Done


http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@16
PS2, Line 16: The issue can be reproduced by creating a Parquet table with many columns and inserting data into it using Hive, then querying the table with Impala. The provided bash and Hive client scripts in IMPALA-12665 create a table and populate it with data to set up the conditions that trigger the bug.
> It would be nice to see a query test covering this case.
Done


http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc@2367
PS2, Line 2367:     complete_micro_batch_.AdjustLength(scratch_batch_->capacity);
> The implementation of AdjustLength can only shrink the micro batch. I'm not
complete_micro_batch_ is used here only and scratch_batch_->capacity in later iterations of this loop won't grow again.



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 07:43:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20834/6/tests/query_test/test_queries.py
File tests/query_test/test_queries.py:

http://gerrit.cloudera.org:8080/#/c/20834/6/tests/query_test/test_queries.py@511
PS6, Line 511: 
flake8: W391 blank line at end of file



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 6
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 08:05:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_queries.py
3 files changed, 52 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 4
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_queries.py
3 files changed, 65 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/8
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 8
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has removed Yifan Zhang from this change.  ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Removed reviewer Yifan Zhang.
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14835/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 1
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Dec 2023 08:01:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 2:

(3 comments)

Thanks for investigating this issue! I have a few comments I'd like to see addressed before merging this.

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
Impala commit messages try to stick to line limits of 72 characters. Not a hard requirement, just lets us format them for easier reading in gerrit. Tends to be more readable in a terminal as well.


http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@16
PS2, Line 16: The issue can be reproduced by creating a Parquet table with many columns and inserting data into it using Hive, then querying the table with Impala. The provided bash and Hive client scripts in IMPALA-12665 create a table and populate it with data to set up the conditions that trigger the bug.
It would be nice to see a query test covering this case.


http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc@2367
PS2, Line 2367:     complete_micro_batch_.AdjustLength(scratch_batch_->capacity);
The implementation of AdjustLength can only shrink the micro batch. I'm not sure there's any guarantee that scratch_batch_->capacity in later iterations of this loop won't grow again. Have you looked into that?



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 2
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 23:24:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 8: Code-Review+1

(3 comments)

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
> Done
Ack


http://gerrit.cloudera.org:8080/#/c/20834/2//COMMIT_MSG@16
PS2, Line 16: The error log by AddressSanitizer points to a heap-buffer-overflow,
> Done
Ack


http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/20834/2/be/src/exec/parquet/hdfs-parquet-scanner.cc@2367
PS2, Line 2367:     complete_micro_batch_.AdjustLength(scratch_batch_->capacity);
> Done
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 8
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 17:07:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14910/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 4
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 07:58:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Zinway (Code Review)" <ge...@cloudera.org>.
Zinway has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................

IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset

**IMPALA-12665 Description:**
The issue occurs when scanning Parquet tables with a row size
> 4096 bytes and a row batch size > 1024. A heap-buffer-overflow
was detected by AddressSanitizer, indicating a write operation
beyond the allocated buffer space.

**Root Cause Analysis:**
The error log by AddressSanitizer points to a heap-buffer-overflow,
where memory is accessed beyond the allocated region. This occurs
in the `HdfsParquetScanner` and `ScratchTupleBatch` classes when
handling large rows > 4096 bytes.

**Fault Reproduction:**
The issue can be reproduced by creating a Parquet table with many
columns, inserting data using Hive, then querying with Impala.
Bash and Hive client scripts in IMPALA-12665 create a table and
populate it, triggering the bug.

**Technical Analysis:**
`ScratchTupleBatch::Reset` recalculates `capacity` based on tuple
size and fixed memory limits. When row size > 4096 bytes, `capacity`
is set < 1024. `HdfsParquetScanner` incorrectly assumes
`complete_micro_batch_` length of 1024, leading to overflow.

**Proposed Solution:**
Ensure `complete_micro_batch_` length is updated after
`ScratchTupleBatch::Reset`. This prevents accessing memory outside
allocated buffer, avoiding heap-buffer-overflow.

Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/scratch-tuple-batch.h
M tests/query_test/test_queries.py
3 files changed, 62 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/20834/7
-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 7
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-12665: Adjust complete micro batch length to new scratch batch ->capacity after ScratchTupleBatch::Reset

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20834 )

Change subject: IMPALA-12665: Adjust complete_micro_batch_ length to new scratch_batch_->capacity after ScratchTupleBatch::Reset
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14912/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I966ff10ba734ed8b1b61325486de0dfcc7b58e4d
Gerrit-Change-Number: 20834
Gerrit-PatchSet: 7
Gerrit-Owner: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zinway <zi...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jan 2024 08:35:52 +0000
Gerrit-HasComments: No