You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2024/03/28 08:43:00 UTC
[jira] [Created] (IMPALA-12949) Need profile counters of BufferPool for UndoMoveEvictedToPinned()
Quanlong Huang created IMPALA-12949:
---------------------------------------
Summary: Need profile counters of BufferPool for UndoMoveEvictedToPinned()
Key: IMPALA-12949
URL: https://issues.apache.org/jira/browse/IMPALA-12949
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
When comparing profiles of two different runs of the same query, I got confused by the BufferPool counters:
{noformat}
Buffer pool | Buffer pool
- AllocTime: 23.74s (23741060380) | - AllocTime: 7.47s (7466380924)
- CompressionTime: 0ns (0) | - CompressionTime: 0ns (0)
- CumulativeAllocationBytes: 32.9GiB (35320410931) | - CumulativeAllocationBytes: 28.6GiB (30664333721)
- CumulativeAllocations: 473,584 (473584) | - CumulativeAllocations: 402,590 (402590)
- EncryptionTime: 0ns (0) | - EncryptionTime: 0ns (0)
- InactiveTotalTime: 0ns (0) | - InactiveTotalTime: 0ns (0)
- PeakReservation: 535.2MiB (561158553) | - PeakReservation: 484.1MiB (507615641)
- PeakUnpinnedBytes: 10.0GiB (10760834252) | - PeakUnpinnedBytes: 10.0GiB (10760939110)
- PeakUsedReservation: 534.8MiB (560798105) | - PeakUsedReservation: 483.9MiB (507419033)
- ReadIoBytes: 9.4GiB (10051446374) | - ReadIoBytes: 5.0GiB (5354822041)
- ReadIoOps: 153,372 (153372) | - ReadIoOps: 81,708 (81708)
- ReadIoWaitTime: 43.1m (2586172694497) | - ReadIoWaitTime: 2.1m (128591149546)
- SystemAllocTime: 21.71s (21707880137) | - SystemAllocTime: 4.75s (4754840861)
- TotalTime: 0ns (0) | - TotalTime: 0ns (0)
- WriteIoBytes: 17.4GiB (18726643302) | - WriteIoBytes: 17.3GiB (18590570905)
- WriteIoOps: 285,745 (285745) | - WriteIoOps: 283,669 (283669)
- WriteIoWaitTime: 36.78s (36777011711) | - WriteIoWaitTime: 1.1m (65635211353){noformat}
WriteIoBytes and WriteIoOps are close. But ReadIoBytes and ReadIoOps on the left are 2x of the right. It seems the operator is spilling the same amount of data but different in reading it back.
Read through the codes, the ReadIo* counters are updated in BufferPool::Client::StartMoveEvictedToPinned() which starts an async read [1]. It can be cancelled by BufferPool::Client::UndoMoveEvictedToPinned(). However, there are no counters showing how many times UndoMoveEvictedToPinned() is invoked. We should add such a counter to explain the difference in ReadIoBytes and ReadIoOps.
[1] https://github.com/apache/impala/blob/c8d56425f87f2bc6da5fd39ed905caeabc9aaad0/be/src/runtime/bufferpool/buffer-pool.cc#L560-L561
--
This message was sent by Atlassian Jira
(v8.20.10#820010)