You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/01/08 19:07:00 UTC
[jira] [Commented] (IMPALA-10048) Speed up dumping breadpad symbols in bin/jenkins/finalize.sh
[ https://issues.apache.org/jira/browse/IMPALA-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804433#comment-17804433 ]
ASF subversion and git services commented on IMPALA-10048:
----------------------------------------------------------
Commit 3bcd770dfc51ed5148cf85a96f5644594f953319 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3bcd770df ]
IMPALA-10048: Go parallel for dump_breakpad_symbols.py
This modifies dump_breakpad_symbols.py to use a ThreadPool
to go parallel when there are multiple binaries or
libraries to process. This is common for Jenkins jobs that
dump symbols for all backend tests. The different binaries
write out to different directories, so the threads don't
interfere with each other.
Testing:
- Ran locally dumping the symbols for all backend tests
- Ran a Jenkins job that generates a minidump and triggers
the minidump symbol processing. It went parallel and
worked fine.
Change-Id: I93427bb07f1d9718bd6df90acfd247210b54294d
Reviewed-on: http://gerrit.cloudera.org:8080/20802
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
> Speed up dumping breadpad symbols in bin/jenkins/finalize.sh
> ------------------------------------------------------------
>
> Key: IMPALA-10048
> URL: https://issues.apache.org/jira/browse/IMPALA-10048
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Minor
> Labels: ramp-up
>
> bin/jenkins/finalize.sh has code to find and symbolize breakpad minidumps. This process is very slow, because it runs bin/dump_breakpad_symbols.py for ${IMPALA_HOME}/be/build/latest, which includes all of the backend tests. There are two problems:
> # It is dumping symbols for a large number of binaries that are not useful unless the minidump is for that particular binary. (i.e. all the backend tests are useless if it is resolving an impalad minidump)
> # bin/dump_breakpad_symbols.py runs serially and could be much faster if it went parallel
> We should fix one or both of these problems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org