You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Yida Wu (Code Review)" <ge...@cloudera.org> on 2024/04/16 17:18:55 UTC

[Impala-ASF-CR] WIP: IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21309


Change subject: WIP: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

WIP: IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new timer counter in the query profile
to track the time spent by scan ranges in the disk io queues.
The solution is to add a timer to each request range and
starting and stopping the timer when the request range is
in the disk io queue. Each request context will have a
queueing timer counter, and every timer associated with
the range belonging to the request context will aggregate
the time count to this counter.

Also adds Reset() function to existing timer to enable
fast resetting of the timer. Since ranges are now reusable,
it is more efficient to reset the timer instead of
reallocating it every time the range is reused.

Tests:
Manuall tests and it shows the aggregated queueing time in
the profile.
TODO: Test the performance impact and add tests.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/util/runtime-profile-counters.h
M be/src/util/stopwatch.h
8 files changed, 90 insertions(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 1
Gerrit-Owner: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new counter TotalDiskIoQueueTime in the query
profile to track the total wall-clock time spent waiting in disk
queues for the specific hdfs scan node.

Instead of using per-range timers and aggregating queueing time
across request ranges, this change uses a shared ConcurrentStopWatch
to measure concurrent wall-clock time while ranges are queued.
Each range starts the timer when it is enqueued and stops it when
dequeued, contributing to the overall wall-clock time, so the metric
reflects the true wall-clock time for the ranges waiting in the disk
queue for this specific hdfs scan node.

Tests:
Added new testcase custom_cluster/test_disk_io_queue.py.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
A tests/custom_cluster/test_disk_io_queue.py
7 files changed, 111 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/6
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 6
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] WIP: IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: WIP: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/15910/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 1
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Apr 2024 17:42:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17285/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 3
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Oct 2024 01:56:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21309/4/be/src/util/stopwatch.h
File be/src/util/stopwatch.h:

http://gerrit.cloudera.org:8080/#/c/21309/4/be/src/util/stopwatch.h@282
PS4, Line 282: , "Reset"
It seems we don't need Reset() here.


http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py
File tests/custom_cluster/test_disk_io_queue.py:

http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py@39
PS4, Line 39:     query = "select * from functional.alltypes limit {0}".format(num_rows)
Can we add test for the case when data_cache is enabled and used by concurrent queries? I'm doing a perf test on S3 with data_cache and concurrent queries so I tested this patch by the way. It seems the counter value is too high:

           - DecompressionTime: 1s209ms
           - MaterializeTupleTime: 4s505ms
           - ScannerIoWaitTime: 1s985ms
           - TotalDiskIoQueueTime: 7s039ms
           - TotalRawHdfsOpenFileTime: 0.000ns
           - TotalRawHdfsReadTime: 267.996ms
           - TotalTime: 8s486ms



-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 4
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Nov 2024 23:56:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new timer counter in the query profile
to track the time spent by scan ranges in the disk io queues.
The solution is to add a timer to each request range and
starting and stopping the timer when the request range is
in the disk io queue. Each request context will have a
queueing timer counter, and every timer associated with
the range belonging to the request context will aggregate
the time count to this counter.

Also adds Reset() function to existing timer to enable
fast resetting of the timer. Since ranges are now reusable,
it is more efficient to reset the timer instead of
reallocating it every time the range is reused.

Tests:
Manuall tests and it shows the aggregated queueing time in
the profile.
Added new testcase custom_cluster/test_disk_io_queue.py.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/util/runtime-profile-counters.h
M be/src/util/stopwatch.h
A tests/custom_cluster/test_disk_io_queue.py
9 files changed, 143 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/4
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 4
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22085/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 7
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 07 Apr 2026 00:08:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py
File tests/custom_cluster/test_disk_io_queue.py:

http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py@39
PS4, Line 39:     query = "select * from functional.alltypes limit {0}".format(num_rows)
> The high value was probably because the counter was aggregating the queue t
Added a test case with data cache enabled and concurrent quereis.



-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 7
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 06 Apr 2026 23:44:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 4
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 31 Oct 2024 16:00:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 5:

Yeah, I can get back to work on this


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 5
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 02 Apr 2026 01:04:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/17283/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Oct 2024 00:42:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new timer counter in the query profile
to track the time spent by scan ranges in the disk io queues.
The solution is to add a timer to each request range and
starting and stopping the timer when the request range is
in the disk io queue. Each request context will have a
queueing timer counter, and every timer associated with
the range belonging to the request context will aggregate
the time count to this counter.

Also adds Reset() function to existing timer to enable
fast resetting of the timer. Since ranges are now reusable,
it is more efficient to reset the timer instead of
reallocating it every time the range is reused.

Tests:
Manuall tests and it shows the aggregated queueing time in
the profile.
Added new testcase custom_cluster/test_disk_io_queue.py.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/util/runtime-profile-counters.h
M be/src/util/stopwatch.h
A tests/custom_cluster/test_disk_io_queue.py
9 files changed, 138 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 2
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22082/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 6
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 06 Apr 2026 18:58:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21309/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21309/7//COMMIT_MSG@16
PS7, Line 16: Each range starts the timer when it is enqueued and stops it when
            : dequeued
The ConcurrentStopWatch is owned by HdfsScanNode and shared by all the corresponding scanner threads. I think it's started when the first scanner adds a request into the queue and stops when there are no concurrent scanner threads waiting for the queue. During this time, some scanner threads might already get the data and are materializing tuples. It's hard to compare this with existing counters like ScannerIoWaitTime, TotalRawHdfsOpenFileTime and TotalRawHdfsReadTime.
https://github.com/apache/impala/blob/de7f610abc4fe7bb74c3ffae5b5369158f4b5eba/be/src/util/stopwatch.h#L207-L222

It'd be more valuable if TotalDiskIoQueueTime is part of ScannerIoWaitTime, i.e., it measures the time when the scanner thread is waiting for data of a ScanRange and the ScanRange is still in the disk queue (HDFS open/read not start yet). And it's the sum of such time in all scanner threads.

I haven't thought about how to implement this yet. But it's more helpful if we can break down ScannerIoWaitTime into TotalRawHdfsOpenFileTime, TotalRawHdfsReadTime,  this counter and other time (probably negligible).



-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 7
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Sun, 12 Apr 2026 14:20:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 4:

Updated the testcase in patch 4 for slight timing variations like 1s999ms. Increased the time to 2.5s for improved stability


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 4
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 31 Oct 2024 15:40:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new timer counter in the query profile
to track the time spent by scan ranges in the disk io queues.
The solution is to add a timer to each request range and
starting and stopping the timer when the request range is
in the disk io queue. Each request context will have a
queueing timer counter, and every timer associated with
the range belonging to the request context will aggregate
the time count to this counter.

Also adds Reset() function to existing timer to enable
fast resetting of the timer. Since ranges are now reusable,
it is more efficient to reset the timer instead of
reallocating it every time the range is reused.

Tests:
Manuall tests and it shows the aggregated queueing time in
the profile.
Added new testcase custom_cluster/test_disk_io_queue.py.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/util/runtime-profile-counters.h
M be/src/util/stopwatch.h
A tests/custom_cluster/test_disk_io_queue.py
9 files changed, 139 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 3
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py
File tests/custom_cluster/test_disk_io_queue.py:

http://gerrit.cloudera.org:8080/#/c/21309/4/tests/custom_cluster/test_disk_io_queue.py@39
PS4, Line 39:     query = "select * from functional.alltypes limit {0}".format(num_rows)
> Can we add test for the case when data_cache is enabled and used by concurr
The high value was probably because the counter was aggregating the queue time for every individual range, meaning overlapping concurrent time. I changed the logic in the latest patch to use ConcurrentStopWatch to capture the true wall-clock time only for the ranges. I will also add the data_cache concurrent test later.



-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 6
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 06 Apr 2026 18:39:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................


Patch Set 5:

I think this is a good addition for observability. It'd be nice to merge it with some solid tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 5
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 02 Apr 2026 00:32:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12046: Add profile counter for scan range queueing time on disk queues

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/21309 )

Change subject: IMPALA-12046: Add profile counter for scan range queueing time on disk queues
......................................................................

IMPALA-12046: Add profile counter for scan range queueing time on disk queues

This patch introduces a new counter TotalDiskIoQueueTime in the query
profile to track the total wall-clock time spent waiting in disk
queues for the specific hdfs scan node.

Instead of using per-range timers and aggregating queueing time
across request ranges, this change uses a shared ConcurrentStopWatch
to measure concurrent wall-clock time while ranges are queued.
Each range starts the timer when it is enqueued and stops it when
dequeued, contributing to the overall wall-clock time, so the metric
reflects the true wall-clock time for the ranges waiting in the disk
queue for this specific hdfs scan node.

Tests:
Added new testcase custom_cluster/test_disk_io_queue.py.

Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
A tests/custom_cluster/test_disk_io_queue.py
7 files changed, 158 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21309/7
-- 
To view, visit http://gerrit.cloudera.org:8080/21309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia0cb24db36cd89933c150b88e166e68a3abc7a60
Gerrit-Change-Number: 21309
Gerrit-PatchSet: 7
Gerrit-Owner: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>