You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Mark Robert Miller (Jira)" <ji...@apache.org> on 2021/09/18 00:50:00 UTC
[jira] [Updated] (LUCENE-10099) Add -Ptests.asyncprofile option.
[ https://issues.apache.org/jira/browse/LUCENE-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Robert Miller updated LUCENE-10099:
----------------------------------------
Description:
Allow async profiler in addition to the JFR profiler.
async profiler description:
{noformat}
This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on the HotSpot JVM.
async-profiler can trace the following kinds of events:
CPU cycles
Hardware and Software performance counters like cache misses, branch misses, page faults, context switches etc.
Allocations in Java Heap
Contented lock attempts, including both Java object monitors and ReentrantLocks
{noformat}
The "tests.asyncprofile" mode switch turns on the async profiler ([https://github.com/jvm-profiling-tools/async-profiler]) during test execution,
and prints a simple summary at the end.
For example, top-10 histogram of methods (cpu samples):
{noformat}
gradlew -p lucene/core test -Ptests.asyncprofile=true{noformat}
Alternatively, you can profile heap allocations instead: TBD
Requirements:
You need to download async profiler 2.x ([https://github.com/jvm-profiling-tools/async-profiler]),
and you need the .so lib in a standard shared folder or one of the directories from LD_LIBRARY_PATH (Linux, though not Arch by
default), DYLD_LIBRARY_PATH (Mac OS), or -Djava.library.path. I think ${HOME}/libs also works
on Linux. We could also allow specifying the location directly as an option.
Debug version of Java is important for best results and is required for heap allocation profiling.
The following permissions need to be set to allow for the proper permissions for profiling:
{noformat}
# sysctl kernel.perf_event_paranoid=1
# sysctl kernel.kptr_restrict=0{noformat}
was:
Man, I don't even remember the last time I contributed an issue to Lucene that was for Lucene. Feels like it has to be near a decade.
I recently finished some work that took quite an investment around automated performance aids. It wasn't giving me that feeling of satisfaction I faintly remember though. Yesterday, I was mostly feeling nostalgic about past times when I would bring up some nasty inefficiency or ugly problem and not hear about how it doesn't likely matter or doesn't sound right or 'meh' - I'd hear "omg, you're right, this must be fixed" and then "you're an idiot, it's even worse than that".
So new little project for today - the first of two things I'd like to contribute - async profiler support for the tests.profile option.
I have a rough draft, it works roughly as tests.profile but instead of:
{noformat}
PROFILE SUMMARY from 174 events (total: 174)
tests.profile.mode=cpu
tests.profile.count=10
tests.profile.stacksize=1
tests.profile.linenumbers=false
PERCENT CPU SAMPLES STACK
5.17% 9 java.util.zip.Inflater#inflateBytesBytes()
3.45% 6 java.io.FileInputStream#open0()
2.30% 4 java.io.UnixFileSystem#getBooleanAttributes0()
2.30% 4 java.io.RandomAccessFile#readBytes()
1.72% 3 java.io.UnixFileSystem#normalize()
1.72% 3 org.apache.lucene.util.automaton.RunAutomaton#<init>()
1.72% 3 java.util.Arrays#fill()
1.15% 2 java.lang.invoke.DirectMethodHandle$Holder#invokeStatic()
1.15% 2 java.lang.StringLatin1#charAt()
1.15% 2 java.io.DataOutputStream#writeUTF()
{noformat}
you get more like:
{noformat}
PROFILE SUMMARY from 368675 events (total: 368675)
tests.profile.mode=cpu
tests.profile.count=10
tests.profile.stacksize=1
tests.profile.linenumbers=false
PERCENT CPU SAMPLES STACK
3.89% 14353 #SymbolTable::lookup_dynamic()
1.67% 6171 #Interpreter()
1.40% 5175 #InstanceKlass::find_method_index()
1.34% 4958 #__tls_get_addr()
1.13% 4177 #I2C/C2I adapters()
1.09% 4022 #resource_allocate_bytes()
0.69% 2528 #ValueStack::values_do()
0.62% 2298 #no_Java_frame()
0.62% 2281 #ScopeDesc::decode_body()
0.55% 2017 #Dictionary::find()
{noformat}
as well as these additional files shown below. Sampling was a bit intense I guess, sorry for the slow load on the flame graph ;)
Summary
https://elated-banach-e02e8b.netlify.app/asyncprofiler/hotspot-pid-686723-id-0-20210912-162040.txt
Results Tree
https://elated-banach-e02e8b.netlify.app/asyncprofiler/tree-hotspot-pid-686723-id-0-20210912-162044.html
Results Flame Graph
https://elated-banach-e02e8b.netlify.app/asyncprofiler/flame-hotspot-pid-686723-id-0-20210912-162040.html
> Add -Ptests.asyncprofile option.
> --------------------------------
>
> Key: LUCENE-10099
> URL: https://issues.apache.org/jira/browse/LUCENE-10099
> Project: Lucene - Core
> Issue Type: Test
> Components: modules/test-framework
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Allow async profiler in addition to the JFR profiler.
> async profiler description:
>
> {noformat}
> This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on the HotSpot JVM.
> async-profiler can trace the following kinds of events:
> CPU cycles
> Hardware and Software performance counters like cache misses, branch misses, page faults, context switches etc.
> Allocations in Java Heap
> Contented lock attempts, including both Java object monitors and ReentrantLocks
> {noformat}
>
> The "tests.asyncprofile" mode switch turns on the async profiler ([https://github.com/jvm-profiling-tools/async-profiler]) during test execution,
> and prints a simple summary at the end.
> For example, top-10 histogram of methods (cpu samples):
> {noformat}
> gradlew -p lucene/core test -Ptests.asyncprofile=true{noformat}
> Alternatively, you can profile heap allocations instead: TBD
> Requirements:
> You need to download async profiler 2.x ([https://github.com/jvm-profiling-tools/async-profiler]),
> and you need the .so lib in a standard shared folder or one of the directories from LD_LIBRARY_PATH (Linux, though not Arch by
> default), DYLD_LIBRARY_PATH (Mac OS), or -Djava.library.path. I think ${HOME}/libs also works
> on Linux. We could also allow specifying the location directly as an option.
> Debug version of Java is important for best results and is required for heap allocation profiling.
> The following permissions need to be set to allow for the proper permissions for profiling:
> {noformat}
> # sysctl kernel.perf_event_paranoid=1
> # sysctl kernel.kptr_restrict=0{noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org