You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/11/06 00:24:00 UTC
[jira] [Commented] (IMPALA-11704) Remote Ozone scans are slow even after data cache warmup
[ https://issues.apache.org/jira/browse/IMPALA-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17629398#comment-17629398 ]
ASF subversion and git services commented on IMPALA-11704:
----------------------------------------------------------
Commit d1d4f183da069b967f7120acfc040e3f6a3598a1 in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d1d4f183d ]
IMPALA-11704: Delay hdfsOpenFile with data cache
Delays hdfsOpenFile until after data cache lookup if using a data cache.
IMPALA-10147 implemented this, but only when using the file handle
cache. This patch adds an additional check in case file handle caching
is disabled.
In networked environments, hdfsOpenFile can take significant time, as
observed in a TPC-DS run of q90 where TotalRawHdfsOpenFileTime
represented a majority of time spent for HDFS_SCAN_NODE. This patch
brings that time to 0 with a primed data cache.
Change-Id: I9429a41fb16de27ccb57730203f95559df0dbfb6
Reviewed-on: http://gerrit.cloudera.org:8080/19204
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Remote Ozone scans are slow even after data cache warmup
> --------------------------------------------------------
>
> Key: IMPALA-11704
> URL: https://issues.apache.org/jira/browse/IMPALA-11704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.1.1
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> From [~drorke]:
> {quote}
> Running some basic performance sanity tests ... with Impala TPC-DS queries against Ozone vs HDFS. Impala appears to be using it's data cache for both Ozone and HDFS remote reads, but in the case of Ozone reads I'm still seeing long scan times and high I/O wait times even after cache warmup. Excerpts below from profiles of q90. Note in both cases the Impala profiles show 100% cache hit rates but for some reason the scan IO wait times are still much longer for the Ozone scans.
> {noformat}
> HDFS:
> - TotalTime: 1s924ms
> - ScannerIoWaitTime: 52.037ms
> Ozone:
> - TotalTime: 8s917ms
> - ScannerIoWaitTime: 7s454ms{noformat}
> If I disable the local cache explicitly via query option I get the following times for the same scan:
> {noformat}
> HDFS:
> - TotalTime: 7s792ms
> - ScannerIoWaitTime: 6s244ms
> Ozone:
> - TotalTime: 8s963ms
> - ScannerIoWaitTime: 7s464ms{noformat}
> {quote}
> Investigating a bit, [~joemcdonnell] noticed in the Ozone profile
> {noformat}
> - ScannerIoWaitTime: 7s454ms
> - TotalRawHdfsOpenFileTime: 5s782ms
> {noformat}
> Based on profile differences around {{TotalRawHdfsOpenFileTime=5s782ms}} (vs {{0ms}} for HDFS), I believe this is a difference in performance when using the data cache but the file handle cache is disabled. That traces back to an incomplete implementation of [IMPALA-10147|https://issues.apache.org/jira/browse/IMPALA-10147].
> A data read:
> 1. [Checks that it can open a file handle|https://github.infra.cloudera.com/CDH/Impala/blob/CDWH-2022.0.10.1/be/src/runtime/io/scan-range.cc#L199]. When file handle cache is enabled, this is a [noop|https://github.infra.cloudera.com/CDH/Impala/blob/CDWH-2022.0.10.1/be/src/runtime/io/hdfs-file-reader.cc#L67].
> 2. It will then try to read data. If data cache is enabled, it will [try to read from the data cache|https://github.infra.cloudera.com/CDH/Impala/blob/CDWH-2022.0.10.1/be/src/runtime/io/hdfs-file-reader.cc#L137].
> 3. If data cache hits, that data is returned and any open file handles are unused.
> When the file handle cache is disabled, opening the file handle [calls hdfsOpenFile and hdfsSeek|https://github.infra.cloudera.com/CDH/Impala/blob/CDWH-2022.0.10.1/be/src/runtime/io/hdfs-file-reader.cc#L70-L72]. {{hdfsOpenFile}} in particular is monitored and added to the profile as {{TotalRawHdfsOpenFileTime}}. That time in the Ozone profile accounts for most of the difference in performance between HDFS and Ozone in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org