You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/05/15 17:09:00 UTC
[jira] [Commented] (IMPALA-8537) Negative values reported for
tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
[ https://issues.apache.org/jira/browse/IMPALA-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840578#comment-16840578 ]
ASF subversion and git services commented on IMPALA-8537:
---------------------------------------------------------
Commit 454d85be8a5b003c725586d2f75ee672925f1962 in impala's branch refs/heads/master from Abhishek
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=454d85b ]
IMPALA-8537: Negative values reported for
tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Whenever closing a FileGroup, the TmpFileMgr::scratch_bytes_used_metric_
was incorrectly being decremented by the total scratch space bytes
across the entire FileGroup
(i.e FileGroup::scratch_space_bytes_used_counter_), for every File in
the FileGroup. This was resulting in the -ive value for the current
scratch space bytes.
The fix is to decrement the TmpFileMgr::scratch_bytes_used_metric_ by
the FileGroup::scratch_space_bytes_used_counter_, only once when the
FileGroup is closed.
Testing Done:
- Added checks for expected current value and HWM of the scratch space
bytes in some of the existing test units in tmp-file-mgr-test.cc.
- Added a new scenario in tmp-file-mgr-test.cc which mimics concurrent
spilling queries and checks for propper current and HWM value for
the scratch space bytes.
- Ad-hoc tests forcing multiple scratch space dirs/files and running
concurrent spilling queries while making sure the current value is
never -ive.
Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4
Reviewed-on: http://gerrit.cloudera.org:8080/13326
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
> --------------------------------------------------------------------------------------------
>
> Key: IMPALA-8537
> URL: https://issues.apache.org/jira/browse/IMPALA-8537
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: David Rorke
> Assignee: Abhishek Rawat
> Priority: Major
> Attachments: bad_spill_metrics.json
>
>
> I'm running a workload that does a lot of spilling and noticed the value reported for tmp-file-mgr.scratch-space-bytes-used is negative on all nodes.
> Some details of the workload and cluster configuration:
> * Generating a 10 TB TPC-DS partitioned parquet data set (very large sort).
> * 30 impalads, each with 48 GB RAM and 14 scratch directories (each on a separate drive)
> * Rough estimate (based on query metrics) of total cumulative aggregate memory spilled across the cluster since restart is 6.5 TB.
> Snapshot of the bad metrics attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org