[jira] [Commented] (IMPALA-8375) Add metrics for spill disk usage

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819702#comment-16819702 ] 

ASF subversion and git services commented on IMPALA-8375:
---------------------------------------------------------

Commit a80ebb7a729c1ffd25db262527d4261e01763646 in impala's branch refs/heads/master from Abhishek Rawat
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a80ebb7 ]

IMPALA-8375: Add metrics for spill disk usage

Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark
& tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current
value for spilled bytes, respectively.

A new class AtomicHighWaterMarkGauge was added to keep track of the HWM
value. The new class also encapsulates a metric object which keeps track
of the current value for the spilled bytes.

The current value is incremented every time a new range is allocated from
a temporary file. The current value for spilled bytes is decremented when
a temporary file is closed. The new metrics are not updated when ranges
are recycled from a file. We can add a new metric in future for keeping
track of actual spilled bytes. The HWM value is updated whenever the
current value is greater than the HWM value.

Testing:
- Added new unit tests to the metrics-test test case.
- E2E testing for both the metrics by running concurrent spilling queries
  and ensuring that both the current value metric and the HWM metric were
  behaving as expected. Ran concurrent queries and monitored the metrics
  on the impala daemon's metric page.

Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298
Reviewed-on: http://gerrit.cloudera.org:8080/12956
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Add metrics for spill disk usage
> --------------------------------
>
>                 Key: IMPALA-8375
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8375
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 3.2.0
>            Reporter: Tim Armstrong
>            Assignee: Abhishek Rawat
>            Priority: Major
>
> * High water mark of bytes allocated
> * Maybe - some metric of utilisation (what % of bytes allocated are actually being used)
> * Maybe - total bytes allocated over the lifetime of the process
> * Maybe - statistics about the sizes of blocks allocated



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org