You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "aldettinger (via GitHub)" <gi...@apache.org> on 2023/10/26 09:46:33 UTC
[I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
aldettinger opened a new issue, #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461
### Describe the feature here
In some cases, it could be relevant to use async-profiler during performance runs.
The installation of async-profile is documented [here](https://github.com/async-profiler/async-profiler#download).
The kernel configuration should be edited as follow:
```
sudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
```
From there, the regression sample app pom file could be completed with a java agent as below:
```
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<executable>java</executable>
<arguments>
<argument>-agentpath:/path-to-async-profiler/lib/libasyncProfiler.so=start,ann,threads,event=cpu,file=report.html</argument>
<argument>-Dquarkus.http.port=${quarkus.http.port}</argument>
<argument>-jar</argument>
<argument>target/quarkus-app/quarkus-run.jar</argument>
</arguments>
</configuration>
```
A `report.html` file is generated in `target/cq-versions-under-test/version`.
Maybe it could be interesting to add a new tool option for this like `-ap path-to-async-profiler`.
Then the pom generation logic should be changed, probably xslt would be a better fit than maven xpp3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "aldettinger (via GitHub)" <gi...@apache.org>.
aldettinger commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1781447927
Thanks @franz1981, `--jfrsync` good to know.
And well done Peter for the test setup @ppalaga.
Yes, the warmup is an open question. As of now, we have caught 1 regression in Quarkus and 2 in Camels without warmup. So it does not feel that bad, probably because warmups are almost same when comparing multiple versions. There are multiple paths that can be taken, improving perf regression investigation, lowering duration, running on prs... Need a bit more thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "franz1981 (via GitHub)" <gi...@apache.org>.
franz1981 commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1780813015
I suggest using jfrSync too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "aldettinger (via GitHub)" <gi...@apache.org>.
aldettinger commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1824583873
Not to forget, it would also be good if the chosen implementation could cover native mode too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "ppalaga (via GitHub)" <gi...@apache.org>.
ppalaga commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1781047059
I worked on something similar in recent days. I took @franz1981 's fabulous [quarkus-profiling-workshop](https://github.com/franz1981/quarkus-profiling-workshop) as a base. The biggest challenge for me was to overcome the unability of wrk/wrk2 to send post requests with a payload. I ended up with using hyperfoil cli and a regular hyperfoil yaml. With those two in place, I was able to attach the profiler only after the warmup was over. I unfortunately cannot share it here, because it contains some customer data.
@aldettinger I see that our cq-perf-regression-scenario.hf.yaml does not have a warmup phase yet. I wonder how useful would the profiling data be without warmup? I wonder whether/how the warmup-profiling coordination could be made through Maven?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "franz1981 (via GitHub)" <gi...@apache.org>.
franz1981 commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1781500848
The problem with short duration is that you need to increase the overhead of the profiler to gather enough sample to be representative enough (eg 1K Hz) but they means increasing the overhead, on likely low powered machines.
If we can assume that we care about macro effects on short running programs on CI (which means few cycles to spare) you can decide to stop at tiered 1 and just avoid the noisy attempts od C2 to compile the "hot" code, which won't likely be hot anyway, but still...if is an A/B test meant to run few times for short time, it's just important that the 2 runs run on similar conditions, and makes sense to reduce the noise due to a constrained environment, but requires some JFK eng to validate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] perf-regression: evaluate the possibility to use async-profiler during runs [camel-quarkus]
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on issue #5461:
URL: https://github.com/apache/camel-quarkus/issues/5461#issuecomment-1781551025
I believe the easiest and safest approach is finding an adequate environment for running this. The best option, IMHO, is running these on dedicated machine (one you can control the full stack of it).
That's what I do for Camel Core. Even though it's not automated or integrated with the contribution lifecycle, the perf tests are open and folks can try to reproduce those if they want to. The downside is that it requires a rather constant observation of what's going on, so you catch these early.
If that's a road you seem feasible, there's some alternatives at our company and I can provide you some pointers to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org