[jira] [Resolved] (IMPALA-5236) Negative byte values are not printed with the correct unit

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-5236.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0


  IMPALA-5158,IMPALA-5236: account for unused buffer pool reservations

We were missing accounting for this, since it is part of the expected
difference between query and process memory consumption. The identity
that applies is is:

  buffers allocated from system =
      reservation + cached buffers - unused reservation

Where "cached buffers" includes free buffers and buffers attached to
clean pages. The reservation is accounted against queries and "buffers
allocated from system" is accounted against the process MemTracker.

Reporting this in a direct way required adding a MemTracker with a
negative consumption consumption, which fortunately did not require
any major changes to the MemTracker code.

Example output when applied to buffer pool branch:

  Process: Limit=8.35 GB Total=579.18 MB Peak=590.41 MB
    Buffer Pool: Free Buffers: Total=268.25 MB
    Buffer Pool: Clean Pages: Total=172.25 MB
    Buffer Pool: Unused Reservation: Total=-8.25 MB
    Free Disk IO Buffers: Total=21.98 MB Peak=21.98 MB
    RequestPool=default-pool: Total=12.07 MB Peak=71.58 MB
      ... <snip> ...
    RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
    Untracked Memory: Total=112.88 MB

Testing:
Added a basic test for MemTrackers with negative metrics.

Change-Id: Idb1fa3110dc893321f9f4e8ced6b7ede12194dad
Reviewed-on: http://gerrit.cloudera.org:8080/7380
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins

> Negative byte values are not printed with the correct unit
> ----------------------------------------------------------
>
>                 Key: IMPALA-5236
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5236
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.9.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Minor
>              Labels: newbie, supportability
>             Fix For: Impala 2.10.0
>
>
> It seems like the logic to decide on the unit does not work correctly for negative values, e.g. it prints "-7369392.00 B" instead of showing MB
> {code}
> [localhost:21000] > select * from tpch.lineitem;
> Query: select * from tpch.lineitem
> Query submitted at: 2017-04-20 12:04:22 (Coordinator: http://localhost:25000)
> Query progress can be monitored at: http://localhost:25000/query_plan?query_id=6048492f67282f78:ef0f2bd400000000
> WARNINGS: Memory limit exceeded: Failed to allocate tuple buffer
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit.
> Error occurred on backend localhost:22000 by fragment 6048492f67282f78:ef0f2bd400000003
> Memory left in process limit: 8.24 GB
> Memory left in query limit: -7369392.00 B
> Query(6048492f67282f78:ef0f2bd400000000): memory limit exceeded. Limit=1.00 MB Total=8.03 MB Peak=8.03 MB
>   Fragment 6048492f67282f78:ef0f2bd400000000: Total=8.00 KB Peak=8.00 KB
>     EXCHANGE_NODE (id=1): Total=0 Peak=0
>     DataStreamRecvr: Total=0 Peak=0
>     PLAN_ROOT_SINK: Total=0 Peak=0
>     CodeGen: Total=0 Peak=0
>   Block Manager: Total=0 Peak=0
>   Fragment 6048492f67282f78:ef0f2bd400000003: Total=8.02 MB Peak=8.02 MB
>     HDFS_SCAN_NODE (id=0): Total=8.01 MB Peak=8.01 MB
>     DataStreamSender (dst_id=1): Total=688.00 B Peak=688.00 B
>     CodeGen: Total=0 Peak=0
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)