You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/01/23 23:28:00 UTC
[jira] [Commented] (IMPALA-12727) Reduce the DiskIoMgr thread parameters for filesystems that are not being tested
[ https://issues.apache.org/jira/browse/IMPALA-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810137#comment-17810137 ]
ASF subversion and git services commented on IMPALA-12727:
----------------------------------------------------------
Commit 3072a2110a558357b328d20ec6bf546ece0251f3 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3072a2110 ]
IMPALA-12727: Reduce IO threads for non-TARGET_FILESYSTEM filesystems
The DiskIoMgr starts a large number of threads for each different
type of object store, most of which are idle. For development,
this slows down processing minidumps and debugging with gdb.
This adds an option "reduce_disk_io_threads" to bin/start-impala-cluster.py
that sets the thread count startup parameter to one for any filesystem
that is not the TARGET_FILESYSTEM. On a typical development setup
running against HDFS, this reduces the number of DiskIoMgr threads
by 150 and the HDFS monitoring threads by 150 as well. This option is
enabled by default. It can disabled by setting --reduce_disk_io_threads=False
for bin/start-impala-cluster.py.
Separately, DiskIoMgr should be modified to reduce the number of
threads it spawns in general.
Testing:
- Hand tested this on my local development system
Change-Id: Ic8ee1fb1f9b9fe65d542d024573562b3bb120b76
Reviewed-on: http://gerrit.cloudera.org:8080/20920
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Reduce the DiskIoMgr thread parameters for filesystems that are not being tested
> --------------------------------------------------------------------------------
>
> Key: IMPALA-12727
> URL: https://issues.apache.org/jira/browse/IMPALA-12727
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Reporter: Joe McDonnell
> Priority: Major
>
> DiskIoMgr is using a large number of threads, most of which are idle. This makes minidumps/cores larger and slower to process. For developers, this can slow down gdb attach and navigating through the threads.
> For our test infrastructure / development, we can reduce the threads for filesystems that are not the TARGET_FILESYSTEM. If we aren't testing ABFS, then we don't need the full set of ABFS threads. In general, we can reduce almost all of the non-TARGET_FILESYSTEM DiskIoMgr thread counts to 1.
> We should have a way to turn it off, but it seems like this could be the default mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org