You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2023/10/18 01:09:00 UTC
[jira] [Commented] (IMPALA-10048) Speed up dumping breadpad symbols in bin/jenkins/finalize.sh
[ https://issues.apache.org/jira/browse/IMPALA-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776439#comment-17776439 ]
Joe McDonnell commented on IMPALA-10048:
----------------------------------------
Probably the simplest way forward is to focus on making bin/dump_breakpad_symbols.py go parallel. Specifically, use a threadpool and run process_binary() in parallel.
See [https://github.com/apache/impala/blob/master/bin/dump_breakpad_symbols.py#L344-L346]
{noformat}
for binary in enumerate_binaries(args):
if not process_binary(dump_syms, objcopy, binary, args.dest_dir):
status = 1{noformat}
We use multiprocessing.pool.ThreadPool in other places like bin/bootstrap_toolchain.py and bin/load-data.py (search for imap_unordered).
> Speed up dumping breadpad symbols in bin/jenkins/finalize.sh
> ------------------------------------------------------------
>
> Key: IMPALA-10048
> URL: https://issues.apache.org/jira/browse/IMPALA-10048
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.0.0
> Reporter: Joe McDonnell
> Priority: Minor
>
> bin/jenkins/finalize.sh has code to find and symbolize breakpad minidumps. This process is very slow, because it runs bin/dump_breakpad_symbols.py for ${IMPALA_HOME}/be/build/latest, which includes all of the backend tests. There are two problems:
> # It is dumping symbols for a large number of binaries that are not useful unless the minidump is for that particular binary. (i.e. all the backend tests are useless if it is resolving an impalad minidump)
> # bin/dump_breakpad_symbols.py runs serially and could be much faster if it went parallel
> We should fix one or both of these problems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org