You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jason Fehr (Code Review)" <ge...@cloudera.org> on 2026/04/01 17:21:46 UTC

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24159


Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. In certain cases, row
batch processing can consume memory until the process memory limit is
reached. At that point, the executor begins failing other queries
with cannotallocated memory errors as they request memory. However,
the row batch processing continues to run since it does no memory
limit checking.

This patch determines the proper soft memory limit based on the
lesser of the query and process soft limit. When expression
evaluation results memory pool exceeds that threshold, the memory
pool is freed during which row batch processing is paused.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M .gitignore
M be/src/codegen/gen_ir_descriptions.py
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
7 files changed, 82 insertions(+), 46 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22601/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 13
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 May 2026 19:15:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 21: Code-Review+2

This looks good to me


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 21
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 May 2026 22:07:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Apr 2026 19:27:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/7/be/src/udf/udf.cc
File be/src/udf/udf.cc:

http://gerrit.cloudera.org:8080/#/c/24159/7/be/src/udf/udf.cc@523
PS7, Line 523:   if (results_pool_max_mem_ > -1 && UNLIKELY(
             :       results_pool_->total_allocated_bytes() + byte_size > results_pool_max_mem_)) {
             :     results_pool_->Clear();
             :   }
How do we know that there are no references to the memory we are clearing? Could I hit this partway through evaluating a conjunct on a single row? For example with conjunct upper(foo) = upper(bar), could I reach this condition when evaluating upper(bar) and clear the result from upper(foo)?



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 May 2026 21:49:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by running the repro query documented in the
Jira with MEM_LIMIT_EXECUTORS=185mb and watching the executor /memz
debug UI page to ensure the query fragment instances stop consuming
memory once they hit their limit.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
3 files changed, 40 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/13
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 13
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed, memory will be consume until
the process memory limit is reached. At that point, the executor
begins failing other queries with cannot allocate memory errors as
they request memory. However, the row batch processing continues to
run since it does no memory limit checking during batch processing.

This patch determines the proper soft memory limit based on the
lesser of the query and process soft limits. When expression
evaluation results memory pool exceeds that threshold, the memory
pool is freed during which row batch processing is paused.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M .gitignore
M be/src/codegen/gen_ir_descriptions.py
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
9 files changed, 144 insertions(+), 50 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 22:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13556/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 22
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 May 2026 22:51:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc
File be/src/exec/partitioned-hash-join-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc@60
PS14, Line 60:       --(*remaining_capacit
> This is basically the right location. The build side can be arbitrarily lar
The latest implementation seems simpler. So I'm not sure where this discussion lands.



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 22:09:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Apr 2026 18:19:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22600/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 May 2026 18:23:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/20/be/src/codegen/llvm-codegen-test.cc
File be/src/codegen/llvm-codegen-test.cc:

http://gerrit.cloudera.org:8080/#/c/24159/20/be/src/codegen/llvm-codegen-test.cc@338
PS20, Line 338:   // Part 2: Replace the default inner looop implementation that does not modify counter
typo: looop



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 22:08:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 22: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 22
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 May 2026 03:27:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc
File be/src/exec/partitioned-hash-join-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc@60
PS14, Line 60:     ClearExprResultsPool();
> Good insight.  I implemented the solution where the call to ClearExprResult
This is basically the right location. The build side can be arbitrarily large, so we can't really move it out to once per probe row. The performance results show that the per-row check is below our noise threshold for TPC-H/TPC-DS. So, maybe we're done with the old version.

The only thought is that maybe codegen is removing this for most TPC-H/TPC-DS queries without any explicit removal from us. So, the question is whether other queries could be impacted.

I don't actually know, but maybe the answer is no. If this condition is always false, speculative execution should look past it. If this condition is happening, other things are more expensive and this is probably not a big deal. Maybe simpler is better here.



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 17:22:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22602/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 May 2026 19:13:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 20:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/22622/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 May 2026 23:41:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 22: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 22
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 May 2026 22:51:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 21: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 21
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 May 2026 06:53:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc
File be/src/exec/partitioned-hash-join-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc@60
PS14, Line 60:       --(*remaining_capacit
> It's possible for a SQL to have a billion duplicates on the build side. Tha
Ah right, thanks for repeating that.



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 22:49:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that
ScalarExprEvaluator instances can consume for their results based on
the lowest soft memory limit in the result MemPool hierarchy
reduced by a factor. When expression evaluation results exceeds
that threshold, the memory pool is cleared during which row batch
processing is paused. Clearing the memory pool does not result in
memory being freed but instead marks all memory in the pool as
available for re-use.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/exec/exec-node.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
5 files changed, 47 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 6
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed, memory will be consume until
the process memory limit is reached. At that point, the executor
begins failing other queries with cannot allocate memory errors as
they request memory. However, the row batch processing continues to
run since it does no memory limit checking during batch processing.

This patch determines the proper soft memory limit based on the
lesser of the query and process soft limits. When expression
evaluation results memory pool exceeds that threshold, the memory
pool is freed during which row batch processing is paused.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M .gitignore
M be/src/codegen/gen_ir_descriptions.py
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
9 files changed, 144 insertions(+), 50 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc
File be/src/exec/partitioned-hash-join-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc@60
PS14, Line 60:     ClearExprResultsPool();
> The latest implementation seems simpler. So I'm not sure where this discuss
It's possible for a SQL to have a billion duplicates on the build side. That means the memory usage for processing a single probe row can still be very large, so I'd prefer an approach that evaluates more often than once per probe row.



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 22:45:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by a new custom cluster test that replicates
the query that revealed this issue.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/partitioned-hash-join-node.inline.h
A tests/custom_cluster/test_conjuncts.py
8 files changed, 251 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/21
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 21
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 20:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22629/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 15:51:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by running the repro query documented in the
Jira with MEM_LIMIT_EXECUTORS=185mb and watching the executor /memz
debug UI page to ensure the query fragment instances stop consuming
memory once they hit their limit.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
3 files changed, 40 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/14
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by running the repro query documented in the
Jira with MEM_LIMIT_EXECUTORS=185mb and watching the executor /memz
debug UI page to ensure the query fragment instances stop consuming
memory once they hit their limit.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
3 files changed, 40 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/12
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 21: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 21
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 23:08:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by a new custom cluster test that replicates
the query that revealed this issue.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Reviewed-on: http://gerrit.cloudera.org:8080/24159
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/partitioned-hash-join-node.inline.h
A tests/custom_cluster/test_conjuncts.py
8 files changed, 251 insertions(+), 3 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 23
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 14:

(1 comment)

The perf-AB-test doesn't show any performance difference.

Is there a way to have an automated test for a problematic SQL?

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc
File be/src/exec/partitioned-hash-join-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/24159/14/be/src/exec/partitioned-hash-join-node-ir.cc@60
PS14, Line 60:     ClearExprResultsPool();
Things push in different directions:

On the one hand, finding a single-row match in the build side hash table is very common. There are lots of joins that have a FK/PK relationship where every probe row matches a single build row. Having a per-row check is a bit overkill for what we are doing. Fewer SQL queries have duplicates on the build side. Skipping calling this on the single-row case would help. If we kept a counter for the for loop, we could call it every few rows.

On the other hand, maybe with codegen we can detect that nothing in the loop will impact expr_results_pool_ (e.g. if there are no other conjuncts) and optimize this away for a lot of SQLs.



-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 May 2026 00:26:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that conjunct
expression evaluation results can consume as a percentage of the
lowest soft memory limit for the query execution. Before evaluating
conjuncts for each join probe row, the expression results memory pool
size is cleared if its allocated memory has exceeded this threshold.
Clearing the memory pool does not result in memory being freed but
instead marks all memory in the pool as available for re-use.

Testing accomplished by a new custom cluster test that replicates
the query that revealed this issue.

Regression functionality testing performed by running existing test
suite. Regression performance testing performed by running AB test
job.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/partitioned-hash-join-node-ir.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/partitioned-hash-join-node.h
M be/src/exec/partitioned-hash-join-node.inline.h
A tests/custom_cluster/test_conjuncts.py
8 files changed, 235 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/20
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 20
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 21:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22644/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 21
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 May 2026 23:27:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22064/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Apr 2026 17:45:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................

[WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Impala checks at the end of every row batch to determine if the query
or process memory limit has been exceeded. If the batch processing
requires evaluating expressions that allocate memory for their
results (such as UPPER() or LOWER()), that memory is allocated
without checking any memory limits. If a large number of these
expression evaluations are performed during a single row batch,
memory will be consumed until the process memory limit is reached.
At that point, the executor begins failing other queries with cannot
allocate memory errors as they request memory. However, the row batch
processing continues to run since it does no memory limit checking
during batch processing.

This patch determines a limit for the amount of memory that
ScalarExprEvaluator instances can consume for their results based on
the lowest soft memory limit in the result MemPool hierarchy
reduced by a factor. When expression evaluation results exceeds
that threshold, the memory pool is cleared during which row batch
processing is paused. Clearing the memory pool does not result in
memory being freed but instead marks all memory in the pool as
available for re-use.

Generated-by: Github Copilot (GPT-5.3-Codex)
Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
---
M be/src/exec/exec-node.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/udf/udf-internal.h
M be/src/udf/udf.cc
5 files changed, 44 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/22276/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 6
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Apr 2026 21:38:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24159 )

Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing
......................................................................


Patch Set 7:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/22277/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24159
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af
Gerrit-Change-Number: 24159
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Apr 2026 22:14:43 +0000
Gerrit-HasComments: No