[jira] [Commented] (IMPALA-11690) Estimate codegen memory usage

["Tamas Mate (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694992#comment-17694992 ] 

Tamas Mate commented on IMPALA-11690:
-------------------------------------

Another way this can happen is a large query with lots of codegen functions but on a small dataset. The estimated memory requirement will be small and the codegen memory request will run out of query memory limit.

> Estimate codegen memory usage
> -----------------------------
>
>                 Key: IMPALA-11690
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11690
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend, Frontend
>    Affects Versions: Impala 4.1.0
>            Reporter: Tamas Mate
>            Priority: Major
>
> Codegen memory requirement is estimated only on the backend before allocation:
> {code:c++}
>   int64_t estimated_memory = ESTIMATED_OPTIMIZER_BYTES_PER_INST
>       * counter.GetCount(InstructionCounter::TOTAL_INSTS);
>   if (!mem_tracker_->TryConsume(estimated_memory)) {
>     const string& msg = Substitute(
>         "Codegen failed to reserve '$0' bytes for optimization", estimated_memory);
>     return mem_tracker_->MemLimitExceeded(NULL, msg, estimated_memory);
>   }
> {code}
> This can cause problems with dedicated coordinator memory estimates where the memory is more tightly estimated, for example:
> {code:none}Query(604397725bdbccbb:8fc7624b00000000): Limit=131.00 MB Reservation=13.94 MB ReservationLimit=99.00 MB OtherMemory=399.55 KB Total=14.33 MB Peak=14.33 MB
>   Unclaimed reservations: Reservation=13.94 MB OtherMemory=0 Total=13.94 MB Peak=13.94 MB
> ...
> Dedicated Coordinator Resource Estimate: Memory=131MB
> {code}
> Impala could apply some heuristics for the codgen memory requirements as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org