You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pekko.apache.org by Matthew de Detrich <ma...@aiven.io.INVALID> on 2024/01/17 01:07:22 UTC
Inliner results for Pekko Http + Dedicated HW for benchmarking
Since the inliner has been merged into both Pekko and Pekko Http I decided
to post some results for pekko-http's jmh bench when all of the inlining
settings are applied (left is without inliner, right is with inliner)
Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
Against Pekko 1.0.0-M1 (current main): https://www.diffchecker.com/XhyEC556/
As you can see from the results there are some noticeable improvements
(i.e. 5-10% in some cases) however I wouldn't take these results as
complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
power plus used TGPro to put fans on max blast to reduce any variability,
unfortunately I am currently overseas so I don't have a dedicated machine
to test on).
This testing was a result of valid concerns raised by Johannes[1].
Another valid point being raised is about being diligent when it comes to
performance/benchmarks which is what dedicated hardware for benchmarking
was meant to solve. PJ Fanning has set up an open collective for Pekko[2]
and there is already a github discussion[3] about it.
So maybe now is a good time to start talking about funding for the Pekko
project in general as well as being a call to action to any heavy users of
Pekko that feel like donating? The initial plan was to use the funds to
procure dedicated HW from hetzner (https://www.hetzner.com/) with dedicated
bare metal machines costing 50-100 euros a month, with one machine being
used for nightly builds and another machine dedicated to benchmarking.
We can also discuss what the additional funds can be used for (i.e.
sponsoring core maintainers, talks/conferences once Pekko gets big enough
etc etc?) although that might be best for another thread.
Thoughts?
[1]:
https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
[2]: https://opencollective.com/friends-of-apache-pekko
[3]: https://github.com/apache/incubator-pekko/discussions/144
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
On Tue, 30 Jan 2024, 21:27 Johannes Rudolph, <jo...@gmail.com>
wrote:
> On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
> <ma...@aiven.io.invalid> wrote:
> >
> > > It will only do that in some select few cases.
> >
> > Agreed, I phrased the original statement incorrectly. I was meant to say
> > that
> > it's not going to pointlessly inline code, i.e. it will only inline code
> > that is known
> > to be not inlinable by earlier JDK's.
>
> This is also not how it works. It has some heuristics about when it
> thinks inlining might at least not hurt and is safe to do, but this is
> pretty much not connected theoretically to any existing JDK
> implementation.
>
> You have seen the diff. 90% of all the optimizer improvements is
> trivial pruning of useless basic blocks and trying to remove redundant
> STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
> works for trivial methods (like the onPull from above). There seem to
> be few places where it works for the intended purpose of optimizing
> HOF for collection usage (sometimes the HOF is inlined but not the
> closure). In other cases where HOF are inlined, these trivial cases
> lead to massive code blow-up which will rather have a negative impact
> on JIT code parsing with the result of the exact same code being
> generated (because these cases are so trivial). In almost every case
> the inliner makes trivial changes that would better be done by the
> JIT.
>
> It's far from clear whether enabling the inliner will create overall
> better code or if it will even put a hit on JIT compilation because of
> the increased code size. It's definitely not a free guaranteed win.
>
It's unfortunate I don't have access to my desktop where I looked at it
mote comprehensively as at the time I didn't notice any regression and I
twayed it with jvm flags that report when the JVM does or doesn't inline.
> > > See https://github.com/apache/incubator-pekko-http/pull/456 for an
> > example of the correctness issue.
> > Afaik none of the many projects enable the inliner for development.
> >
> > By correctness I was referring to the inliner producing incorrect code
> on a
> > clean
> > compile.
>
> I guess our major disagreement is about priorities. I highly
> prioritize my time as a (for this project mostly unpaid) developer. A
> 2x-3x hit on compilation times is huge in such a big project that
> often accidentally triggers major recompilation.
>
> If you think about it in general, it's far more important for this
> project to remove hurdles for potential contributors than to squeeze
> out the last bits of performance (which is unclear anyway).
>
I admitted earlier that the current default is not ideal and have zero
qualms in changing it.
---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> I'm talking about pekkoInlineEnabled. There is discussion about only
enabling this in CI. Releases are not done using CI. So when we release
pekkoInlineEnabled will be false. It makes no sense to me to enable
pekkoInlineEnabled at all if we don't intend the Pekko 1.1.0-M1 to have it
enabled. It makes our snapshots where pekkoInlineEnabled=true different
from the likely 1.1.0-M1 release.
Agreed, this was a miscommunication then. If we only
have pekkoInlineEnabled in CI then we have to adjust the release notes so
that pekkoInlineEnabled is also enabled when making release artifacts
On Thu, Feb 1, 2024 at 12:01 AM PJ Fanning <fa...@apache.org> wrote:
> I'm talking about pekkoInlineEnabled. There is discussion about only
> enabling this in CI. Releases are not done using CI. So when we release
> pekkoInlineEnabled will be false. It makes no sense to me to enable
> pekkoInlineEnabled at all if we don't intend the Pekko 1.1.0-M1 to have it
> enabled. It makes our snapshots where pekkoInlineEnabled=true different
> from the likely 1.1.0-M1 release.
>
> On 2024/01/31 07:50:48 Matthew de Detrich wrote:
> > > If everyone feels that there may be risks with inline, can we add a
> > separate inline test in ci? Or even release a separate inline jar
> > specifically for daily SNAPSHOT preview versions. We can even write a
> test
> > in downstream projects such as pekko-http to test the performance
> > difference between inline and regular versions.
> >
> > I don't have an issue doing this to placate concerns but to be blunt
> since
> > there has been zero evidence of any actual risks (the correctness
> concerns
> > alluded to earlier was a miscommunication) I do see it as a worse middle
> > ground option that wouldn't alleviate anyones concerns from either side.
> >
> > As a related tidbit, one of the motivated reasons for me having the
> inliner
> > enabled by default locally (which re-iterating I now regret) was to help
> > weed out any potential issues, i.e. lower the risk
> >
> > > The main problem now is
> > that we can only prove that there are no problems with the parts covered
> by
> > our tests, but we really have no way to prove that there are no problems
> > with the parts not covered by our tests :(
> >
> > The coverage of our test suite is quite extensive especially considering
> > that we are extending an entire matrix to all of our projects now.
> However
> > what I want to get to is that the main point of releasing M1 with the
> > inliner
> > enabled is to do exactly that, i.e. get people to run it in more battle
> > tested
> > scenarios.
> >
> > If there is a problem we revert it, now that we have sbt-pekko-build its
> > fairly trivial to do so.
> >
> > On Wed, Jan 31, 2024 at 5:33 PM kerr <he...@gmail.com> wrote:
> >
> > > If everyone feels that there may be risks with inline, can we add a
> > > separate inline test in ci? Or even release a separate inline jar
> > > specifically for daily SNAPSHOT preview versions. We can even write a
> test
> > > in downstream projects such as pekko-http to test the performance
> > > difference between inline and regular versions. The main problem now is
> > > that we can only prove that there are no problems with the parts
> covered by
> > > our tests, but we really have no way to prove that there are no
> problems
> > > with the parts not covered by our tests :(
> > >
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by PJ Fanning <fa...@apache.org>.
I'm talking about pekkoInlineEnabled. There is discussion about only enabling this in CI. Releases are not done using CI. So when we release pekkoInlineEnabled will be false. It makes no sense to me to enable pekkoInlineEnabled at all if we don't intend the Pekko 1.1.0-M1 to have it enabled. It makes our snapshots where pekkoInlineEnabled=true different from the likely 1.1.0-M1 release.
On 2024/01/31 07:50:48 Matthew de Detrich wrote:
> > If everyone feels that there may be risks with inline, can we add a
> separate inline test in ci? Or even release a separate inline jar
> specifically for daily SNAPSHOT preview versions. We can even write a test
> in downstream projects such as pekko-http to test the performance
> difference between inline and regular versions.
>
> I don't have an issue doing this to placate concerns but to be blunt since
> there has been zero evidence of any actual risks (the correctness concerns
> alluded to earlier was a miscommunication) I do see it as a worse middle
> ground option that wouldn't alleviate anyones concerns from either side.
>
> As a related tidbit, one of the motivated reasons for me having the inliner
> enabled by default locally (which re-iterating I now regret) was to help
> weed out any potential issues, i.e. lower the risk
>
> > The main problem now is
> that we can only prove that there are no problems with the parts covered by
> our tests, but we really have no way to prove that there are no problems
> with the parts not covered by our tests :(
>
> The coverage of our test suite is quite extensive especially considering
> that we are extending an entire matrix to all of our projects now. However
> what I want to get to is that the main point of releasing M1 with the
> inliner
> enabled is to do exactly that, i.e. get people to run it in more battle
> tested
> scenarios.
>
> If there is a problem we revert it, now that we have sbt-pekko-build its
> fairly trivial to do so.
>
> On Wed, Jan 31, 2024 at 5:33 PM kerr <he...@gmail.com> wrote:
>
> > If everyone feels that there may be risks with inline, can we add a
> > separate inline test in ci? Or even release a separate inline jar
> > specifically for daily SNAPSHOT preview versions. We can even write a test
> > in downstream projects such as pekko-http to test the performance
> > difference between inline and regular versions. The main problem now is
> > that we can only prove that there are no problems with the parts covered by
> > our tests, but we really have no way to prove that there are no problems
> > with the parts not covered by our tests :(
> >
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
For additional commands, e-mail: dev-help@pekko.apache.org
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> If everyone feels that there may be risks with inline, can we add a
separate inline test in ci? Or even release a separate inline jar
specifically for daily SNAPSHOT preview versions. We can even write a test
in downstream projects such as pekko-http to test the performance
difference between inline and regular versions.
I don't have an issue doing this to placate concerns but to be blunt since
there has been zero evidence of any actual risks (the correctness concerns
alluded to earlier was a miscommunication) I do see it as a worse middle
ground option that wouldn't alleviate anyones concerns from either side.
As a related tidbit, one of the motivated reasons for me having the inliner
enabled by default locally (which re-iterating I now regret) was to help
weed out any potential issues, i.e. lower the risk
> The main problem now is
that we can only prove that there are no problems with the parts covered by
our tests, but we really have no way to prove that there are no problems
with the parts not covered by our tests :(
The coverage of our test suite is quite extensive especially considering
that we are extending an entire matrix to all of our projects now. However
what I want to get to is that the main point of releasing M1 with the
inliner
enabled is to do exactly that, i.e. get people to run it in more battle
tested
scenarios.
If there is a problem we revert it, now that we have sbt-pekko-build its
fairly trivial to do so.
On Wed, Jan 31, 2024 at 5:33 PM kerr <he...@gmail.com> wrote:
> If everyone feels that there may be risks with inline, can we add a
> separate inline test in ci? Or even release a separate inline jar
> specifically for daily SNAPSHOT preview versions. We can even write a test
> in downstream projects such as pekko-http to test the performance
> difference between inline and regular versions. The main problem now is
> that we can only prove that there are no problems with the parts covered by
> our tests, but we really have no way to prove that there are no problems
> with the parts not covered by our tests :(
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
If everyone feels that there may be risks with inline, can we add a
separate inline test in ci? Or even release a separate inline jar
specifically for daily SNAPSHOT preview versions. We can even write a test
in downstream projects such as pekko-http to test the performance
difference between inline and regular versions. The main problem now is
that we can only prove that there are no problems with the parts covered by
our tests, but we really have no way to prove that there are no problems
with the parts not covered by our tests :(
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> What's the result jar size, inline vs noinline.
I posted results in one of the gitub issues/PRs and it was less than 1%
bigger
On Wed, 31 Jan 2024, 16:31 kerr, <he...@gmail.com> wrote:
> What's the result jar size, inline vs noinline.
> 何品
>
>
> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月31日周三
> 09:01写道：
>
> > > Actually what's been happening is the exact opposite, all of our
> testing
> > for M1 is being done on the inlined variant and not on the inline one.
> >
> > I meant to say "not on the non-inlined one"
> >
> > On Wed, 31 Jan 2024, 09:17 Matthew de Detrich, <
> matthew.dedetrich@aiven.io
> > >
> > wrote:
> >
> > >
> > >
> > > On Tue, 30 Jan 2024, 23:14 PJ Fanning, <fa...@apache.org> wrote:
> > >
> > >> I would favour disabling the inlining until at least after the
> 1.1.0-M1
> > >> release.
> > >>
> > >> Things are moving to the stage where we don't want to enable inlining
> > >> except in CI builds. Our releases are not done using CI. The releases
> > >> are not going to be automated for a while due the need for us to use
> > >> developer keys and the fact that we can't put those in a CI env. If
> > >> our releases can't have inlining, I don't see why our CI builds,
> > >> including the publishing of our snapshot jars, should have inlining.
> > >>
> > >
> > > Actually what's been happening is the exact opposite, all of our
> testing
> > > for M1 is being done on the inlined variant and not on the inline one.
> I
> > am
> > > also running both pekko and pekko-http nightly snapshots in some
> systems
> > > and haven't noticed any regression. So suddenly releasing with a non
> > > inlined M1 is a drastic change.
> > >
> > > If you referring to the Scala 3 inline keyword that's an entirely
> > > different topic/problem.
> > >
> > > So changing it now is actually the more drastic option and the main
> > reason
> > > for adding the inliner in published snapshots earlier was so extensive
> > > testing could be done on it. The last thing I want to do is to just
> > enabled
> > > inliner last minute when release is occurring.
> > >
> > > I don't have an issue in reverting the change, I am saying let's do it
> > > when it's clear there is an obvious issue. The issue that Johannes
> > pointed
> > > out with local dev is a clear one that I agree with and I have already
> > been
> > > doing this change in other projects.
> > >
> > >
> > >> At the end of the day, I think we should wait until we can automate
> > >> the releases until we consider adding back inlining.
> > >>
> > >> On Tue, 30 Jan 2024 at 11:27, Johannes Rudolph
> > >> <jo...@gmail.com> wrote:
> > >> >
> > >> > On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
> > >> > <ma...@aiven.io.invalid> wrote:
> > >> > >
> > >> > > > It will only do that in some select few cases.
> > >> > >
> > >> > > Agreed, I phrased the original statement incorrectly. I was meant
> to
> > >> say
> > >> > > that
> > >> > > it's not going to pointlessly inline code, i.e. it will only
> inline
> > >> code
> > >> > > that is known
> > >> > > to be not inlinable by earlier JDK's.
> > >> >
> > >> > This is also not how it works. It has some heuristics about when it
> > >> > thinks inlining might at least not hurt and is safe to do, but this
> is
> > >> > pretty much not connected theoretically to any existing JDK
> > >> > implementation.
> > >> >
> > >> > You have seen the diff. 90% of all the optimizer improvements is
> > >> > trivial pruning of useless basic blocks and trying to remove
> redundant
> > >> > STORE/LOAD pairs. Then there's a tiny bit of extra inlining which
> only
> > >> > works for trivial methods (like the onPull from above). There seem
> to
> > >> > be few places where it works for the intended purpose of optimizing
> > >> > HOF for collection usage (sometimes the HOF is inlined but not the
> > >> > closure). In other cases where HOF are inlined, these trivial cases
> > >> > lead to massive code blow-up which will rather have a negative
> impact
> > >> > on JIT code parsing with the result of the exact same code being
> > >> > generated (because these cases are so trivial). In almost every case
> > >> > the inliner makes trivial changes that would better be done by the
> > >> > JIT.
> > >> >
> > >> > It's far from clear whether enabling the inliner will create overall
> > >> > better code or if it will even put a hit on JIT compilation because
> of
> > >> > the increased code size. It's definitely not a free guaranteed win.
> > >> >
> > >> > > > See https://github.com/apache/incubator-pekko-http/pull/456 for
> > an
> > >> > > example of the correctness issue.
> > >> > > Afaik none of the many projects enable the inliner for
> development.
> > >> > >
> > >> > > By correctness I was referring to the inliner producing incorrect
> > >> code on a
> > >> > > clean
> > >> > > compile.
> > >> >
> > >> > I guess our major disagreement is about priorities. I highly
> > >> > prioritize my time as a (for this project mostly unpaid) developer.
> A
> > >> > 2x-3x hit on compilation times is huge in such a big project that
> > >> > often accidentally triggers major recompilation.
> > >> >
> > >> > If you think about it in general, it's far more important for this
> > >> > project to remove hurdles for potential contributors than to squeeze
> > >> > out the last bits of performance (which is unclear anyway).
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> > >> > For additional commands, e-mail: dev-help@pekko.apache.org
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> > >> For additional commands, e-mail: dev-help@pekko.apache.org
> > >>
> > >>
> >
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
What's the result jar size, inline vs noinline.
何品
Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月31日周三
09:01写道：
> > Actually what's been happening is the exact opposite, all of our testing
> for M1 is being done on the inlined variant and not on the inline one.
>
> I meant to say "not on the non-inlined one"
>
> On Wed, 31 Jan 2024, 09:17 Matthew de Detrich, <matthew.dedetrich@aiven.io
> >
> wrote:
>
> >
> >
> > On Tue, 30 Jan 2024, 23:14 PJ Fanning, <fa...@apache.org> wrote:
> >
> >> I would favour disabling the inlining until at least after the 1.1.0-M1
> >> release.
> >>
> >> Things are moving to the stage where we don't want to enable inlining
> >> except in CI builds. Our releases are not done using CI. The releases
> >> are not going to be automated for a while due the need for us to use
> >> developer keys and the fact that we can't put those in a CI env. If
> >> our releases can't have inlining, I don't see why our CI builds,
> >> including the publishing of our snapshot jars, should have inlining.
> >>
> >
> > Actually what's been happening is the exact opposite, all of our testing
> > for M1 is being done on the inlined variant and not on the inline one. I
> am
> > also running both pekko and pekko-http nightly snapshots in some systems
> > and haven't noticed any regression. So suddenly releasing with a non
> > inlined M1 is a drastic change.
> >
> > If you referring to the Scala 3 inline keyword that's an entirely
> > different topic/problem.
> >
> > So changing it now is actually the more drastic option and the main
> reason
> > for adding the inliner in published snapshots earlier was so extensive
> > testing could be done on it. The last thing I want to do is to just
> enabled
> > inliner last minute when release is occurring.
> >
> > I don't have an issue in reverting the change, I am saying let's do it
> > when it's clear there is an obvious issue. The issue that Johannes
> pointed
> > out with local dev is a clear one that I agree with and I have already
> been
> > doing this change in other projects.
> >
> >
> >> At the end of the day, I think we should wait until we can automate
> >> the releases until we consider adding back inlining.
> >>
> >> On Tue, 30 Jan 2024 at 11:27, Johannes Rudolph
> >> <jo...@gmail.com> wrote:
> >> >
> >> > On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
> >> > <ma...@aiven.io.invalid> wrote:
> >> > >
> >> > > > It will only do that in some select few cases.
> >> > >
> >> > > Agreed, I phrased the original statement incorrectly. I was meant to
> >> say
> >> > > that
> >> > > it's not going to pointlessly inline code, i.e. it will only inline
> >> code
> >> > > that is known
> >> > > to be not inlinable by earlier JDK's.
> >> >
> >> > This is also not how it works. It has some heuristics about when it
> >> > thinks inlining might at least not hurt and is safe to do, but this is
> >> > pretty much not connected theoretically to any existing JDK
> >> > implementation.
> >> >
> >> > You have seen the diff. 90% of all the optimizer improvements is
> >> > trivial pruning of useless basic blocks and trying to remove redundant
> >> > STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
> >> > works for trivial methods (like the onPull from above). There seem to
> >> > be few places where it works for the intended purpose of optimizing
> >> > HOF for collection usage (sometimes the HOF is inlined but not the
> >> > closure). In other cases where HOF are inlined, these trivial cases
> >> > lead to massive code blow-up which will rather have a negative impact
> >> > on JIT code parsing with the result of the exact same code being
> >> > generated (because these cases are so trivial). In almost every case
> >> > the inliner makes trivial changes that would better be done by the
> >> > JIT.
> >> >
> >> > It's far from clear whether enabling the inliner will create overall
> >> > better code or if it will even put a hit on JIT compilation because of
> >> > the increased code size. It's definitely not a free guaranteed win.
> >> >
> >> > > > See https://github.com/apache/incubator-pekko-http/pull/456 for
> an
> >> > > example of the correctness issue.
> >> > > Afaik none of the many projects enable the inliner for development.
> >> > >
> >> > > By correctness I was referring to the inliner producing incorrect
> >> code on a
> >> > > clean
> >> > > compile.
> >> >
> >> > I guess our major disagreement is about priorities. I highly
> >> > prioritize my time as a (for this project mostly unpaid) developer. A
> >> > 2x-3x hit on compilation times is huge in such a big project that
> >> > often accidentally triggers major recompilation.
> >> >
> >> > If you think about it in general, it's far more important for this
> >> > project to remove hurdles for potential contributors than to squeeze
> >> > out the last bits of performance (which is unclear anyway).
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> >> > For additional commands, e-mail: dev-help@pekko.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> >> For additional commands, e-mail: dev-help@pekko.apache.org
> >>
> >>
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> Actually what's been happening is the exact opposite, all of our testing
for M1 is being done on the inlined variant and not on the inline one.
I meant to say "not on the non-inlined one"
On Wed, 31 Jan 2024, 09:17 Matthew de Detrich, <ma...@aiven.io>
wrote:
>
>
> On Tue, 30 Jan 2024, 23:14 PJ Fanning, <fa...@apache.org> wrote:
>
>> I would favour disabling the inlining until at least after the 1.1.0-M1
>> release.
>>
>> Things are moving to the stage where we don't want to enable inlining
>> except in CI builds. Our releases are not done using CI. The releases
>> are not going to be automated for a while due the need for us to use
>> developer keys and the fact that we can't put those in a CI env. If
>> our releases can't have inlining, I don't see why our CI builds,
>> including the publishing of our snapshot jars, should have inlining.
>>
>
> Actually what's been happening is the exact opposite, all of our testing
> for M1 is being done on the inlined variant and not on the inline one. I am
> also running both pekko and pekko-http nightly snapshots in some systems
> and haven't noticed any regression. So suddenly releasing with a non
> inlined M1 is a drastic change.
>
> If you referring to the Scala 3 inline keyword that's an entirely
> different topic/problem.
>
> So changing it now is actually the more drastic option and the main reason
> for adding the inliner in published snapshots earlier was so extensive
> testing could be done on it. The last thing I want to do is to just enabled
> inliner last minute when release is occurring.
>
> I don't have an issue in reverting the change, I am saying let's do it
> when it's clear there is an obvious issue. The issue that Johannes pointed
> out with local dev is a clear one that I agree with and I have already been
> doing this change in other projects.
>
>
>> At the end of the day, I think we should wait until we can automate
>> the releases until we consider adding back inlining.
>>
>> On Tue, 30 Jan 2024 at 11:27, Johannes Rudolph
>> <jo...@gmail.com> wrote:
>> >
>> > On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
>> > <ma...@aiven.io.invalid> wrote:
>> > >
>> > > > It will only do that in some select few cases.
>> > >
>> > > Agreed, I phrased the original statement incorrectly. I was meant to
>> say
>> > > that
>> > > it's not going to pointlessly inline code, i.e. it will only inline
>> code
>> > > that is known
>> > > to be not inlinable by earlier JDK's.
>> >
>> > This is also not how it works. It has some heuristics about when it
>> > thinks inlining might at least not hurt and is safe to do, but this is
>> > pretty much not connected theoretically to any existing JDK
>> > implementation.
>> >
>> > You have seen the diff. 90% of all the optimizer improvements is
>> > trivial pruning of useless basic blocks and trying to remove redundant
>> > STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
>> > works for trivial methods (like the onPull from above). There seem to
>> > be few places where it works for the intended purpose of optimizing
>> > HOF for collection usage (sometimes the HOF is inlined but not the
>> > closure). In other cases where HOF are inlined, these trivial cases
>> > lead to massive code blow-up which will rather have a negative impact
>> > on JIT code parsing with the result of the exact same code being
>> > generated (because these cases are so trivial). In almost every case
>> > the inliner makes trivial changes that would better be done by the
>> > JIT.
>> >
>> > It's far from clear whether enabling the inliner will create overall
>> > better code or if it will even put a hit on JIT compilation because of
>> > the increased code size. It's definitely not a free guaranteed win.
>> >
>> > > > See https://github.com/apache/incubator-pekko-http/pull/456 for an
>> > > example of the correctness issue.
>> > > Afaik none of the many projects enable the inliner for development.
>> > >
>> > > By correctness I was referring to the inliner producing incorrect
>> code on a
>> > > clean
>> > > compile.
>> >
>> > I guess our major disagreement is about priorities. I highly
>> > prioritize my time as a (for this project mostly unpaid) developer. A
>> > 2x-3x hit on compilation times is huge in such a big project that
>> > often accidentally triggers major recompilation.
>> >
>> > If you think about it in general, it's far more important for this
>> > project to remove hurdles for potential contributors than to squeeze
>> > out the last bits of performance (which is unclear anyway).
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
>> > For additional commands, e-mail: dev-help@pekko.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
>> For additional commands, e-mail: dev-help@pekko.apache.org
>>
>>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
On Tue, 30 Jan 2024, 23:14 PJ Fanning, <fa...@apache.org> wrote:
> I would favour disabling the inlining until at least after the 1.1.0-M1
> release.
>
> Things are moving to the stage where we don't want to enable inlining
> except in CI builds. Our releases are not done using CI. The releases
> are not going to be automated for a while due the need for us to use
> developer keys and the fact that we can't put those in a CI env. If
> our releases can't have inlining, I don't see why our CI builds,
> including the publishing of our snapshot jars, should have inlining.
>
Actually what's been happening is the exact opposite, all of our testing
for M1 is being done on the inlined variant and not on the inline one. I am
also running both pekko and pekko-http nightly snapshots in some systems
and haven't noticed any regression. So suddenly releasing with a non
inlined M1 is a drastic change.
If you referring to the Scala 3 inline keyword that's an entirely different
topic/problem.
So changing it now is actually the more drastic option and the main reason
for adding the inliner in published snapshots earlier was so extensive
testing could be done on it. The last thing I want to do is to just enabled
inliner last minute when release is occurring.
I don't have an issue in reverting the change, I am saying let's do it when
it's clear there is an obvious issue. The issue that Johannes pointed out
with local dev is a clear one that I agree with and I have already been
doing this change in other projects.
> At the end of the day, I think we should wait until we can automate
> the releases until we consider adding back inlining.
>
> On Tue, 30 Jan 2024 at 11:27, Johannes Rudolph
> <jo...@gmail.com> wrote:
> >
> > On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
> > <ma...@aiven.io.invalid> wrote:
> > >
> > > > It will only do that in some select few cases.
> > >
> > > Agreed, I phrased the original statement incorrectly. I was meant to
> say
> > > that
> > > it's not going to pointlessly inline code, i.e. it will only inline
> code
> > > that is known
> > > to be not inlinable by earlier JDK's.
> >
> > This is also not how it works. It has some heuristics about when it
> > thinks inlining might at least not hurt and is safe to do, but this is
> > pretty much not connected theoretically to any existing JDK
> > implementation.
> >
> > You have seen the diff. 90% of all the optimizer improvements is
> > trivial pruning of useless basic blocks and trying to remove redundant
> > STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
> > works for trivial methods (like the onPull from above). There seem to
> > be few places where it works for the intended purpose of optimizing
> > HOF for collection usage (sometimes the HOF is inlined but not the
> > closure). In other cases where HOF are inlined, these trivial cases
> > lead to massive code blow-up which will rather have a negative impact
> > on JIT code parsing with the result of the exact same code being
> > generated (because these cases are so trivial). In almost every case
> > the inliner makes trivial changes that would better be done by the
> > JIT.
> >
> > It's far from clear whether enabling the inliner will create overall
> > better code or if it will even put a hit on JIT compilation because of
> > the increased code size. It's definitely not a free guaranteed win.
> >
> > > > See https://github.com/apache/incubator-pekko-http/pull/456 for an
> > > example of the correctness issue.
> > > Afaik none of the many projects enable the inliner for development.
> > >
> > > By correctness I was referring to the inliner producing incorrect code
> on a
> > > clean
> > > compile.
> >
> > I guess our major disagreement is about priorities. I highly
> > prioritize my time as a (for this project mostly unpaid) developer. A
> > 2x-3x hit on compilation times is huge in such a big project that
> > often accidentally triggers major recompilation.
> >
> > If you think about it in general, it's far more important for this
> > project to remove hurdles for potential contributors than to squeeze
> > out the last bits of performance (which is unclear anyway).
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> > For additional commands, e-mail: dev-help@pekko.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by PJ Fanning <fa...@apache.org>.
I would favour disabling the inlining until at least after the 1.1.0-M1 release.
Things are moving to the stage where we don't want to enable inlining
except in CI builds. Our releases are not done using CI. The releases
are not going to be automated for a while due the need for us to use
developer keys and the fact that we can't put those in a CI env. If
our releases can't have inlining, I don't see why our CI builds,
including the publishing of our snapshot jars, should have inlining.
At the end of the day, I think we should wait until we can automate
the releases until we consider adding back inlining.
On Tue, 30 Jan 2024 at 11:27, Johannes Rudolph
<jo...@gmail.com> wrote:
>
> On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
> <ma...@aiven.io.invalid> wrote:
> >
> > > It will only do that in some select few cases.
> >
> > Agreed, I phrased the original statement incorrectly. I was meant to say
> > that
> > it's not going to pointlessly inline code, i.e. it will only inline code
> > that is known
> > to be not inlinable by earlier JDK's.
>
> This is also not how it works. It has some heuristics about when it
> thinks inlining might at least not hurt and is safe to do, but this is
> pretty much not connected theoretically to any existing JDK
> implementation.
>
> You have seen the diff. 90% of all the optimizer improvements is
> trivial pruning of useless basic blocks and trying to remove redundant
> STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
> works for trivial methods (like the onPull from above). There seem to
> be few places where it works for the intended purpose of optimizing
> HOF for collection usage (sometimes the HOF is inlined but not the
> closure). In other cases where HOF are inlined, these trivial cases
> lead to massive code blow-up which will rather have a negative impact
> on JIT code parsing with the result of the exact same code being
> generated (because these cases are so trivial). In almost every case
> the inliner makes trivial changes that would better be done by the
> JIT.
>
> It's far from clear whether enabling the inliner will create overall
> better code or if it will even put a hit on JIT compilation because of
> the increased code size. It's definitely not a free guaranteed win.
>
> > > See https://github.com/apache/incubator-pekko-http/pull/456 for an
> > example of the correctness issue.
> > Afaik none of the many projects enable the inliner for development.
> >
> > By correctness I was referring to the inliner producing incorrect code on a
> > clean
> > compile.
>
> I guess our major disagreement is about priorities. I highly
> prioritize my time as a (for this project mostly unpaid) developer. A
> 2x-3x hit on compilation times is huge in such a big project that
> often accidentally triggers major recompilation.
>
> If you think about it in general, it's far more important for this
> project to remove hurdles for potential contributors than to squeeze
> out the last bits of performance (which is unclear anyway).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
For additional commands, e-mail: dev-help@pekko.apache.org
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Johannes Rudolph <jo...@gmail.com>.
On Tue, Jan 30, 2024 at 8:30 AM Matthew de Detrich
<ma...@aiven.io.invalid> wrote:
>
> > It will only do that in some select few cases.
>
> Agreed, I phrased the original statement incorrectly. I was meant to say
> that
> it's not going to pointlessly inline code, i.e. it will only inline code
> that is known
> to be not inlinable by earlier JDK's.
This is also not how it works. It has some heuristics about when it
thinks inlining might at least not hurt and is safe to do, but this is
pretty much not connected theoretically to any existing JDK
implementation.
You have seen the diff. 90% of all the optimizer improvements is
trivial pruning of useless basic blocks and trying to remove redundant
STORE/LOAD pairs. Then there's a tiny bit of extra inlining which only
works for trivial methods (like the onPull from above). There seem to
be few places where it works for the intended purpose of optimizing
HOF for collection usage (sometimes the HOF is inlined but not the
closure). In other cases where HOF are inlined, these trivial cases
lead to massive code blow-up which will rather have a negative impact
on JIT code parsing with the result of the exact same code being
generated (because these cases are so trivial). In almost every case
the inliner makes trivial changes that would better be done by the
JIT.
It's far from clear whether enabling the inliner will create overall
better code or if it will even put a hit on JIT compilation because of
the increased code size. It's definitely not a free guaranteed win.
> > See https://github.com/apache/incubator-pekko-http/pull/456 for an
> example of the correctness issue.
> Afaik none of the many projects enable the inliner for development.
>
> By correctness I was referring to the inliner producing incorrect code on a
> clean
> compile.
I guess our major disagreement is about priorities. I highly
prioritize my time as a (for this project mostly unpaid) developer. A
2x-3x hit on compilation times is huge in such a big project that
often accidentally triggers major recompilation.
If you think about it in general, it's far more important for this
project to remove hurdles for potential contributors than to squeeze
out the last bits of performance (which is unclear anyway).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
For additional commands, e-mail: dev-help@pekko.apache.org
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> It will only do that in some select few cases.
Agreed, I phrased the original statement incorrectly. I was meant to say
that
it's not going to pointlessly inline code, i.e. it will only inline code
that is known
to be not inlinable by earlier JDK's. Its of course impossible to inline
any/all
hypothetical code
> See https://github.com/apache/incubator-pekko-http/pull/456 for an
example of the correctness issue.
Afaik none of the many projects enable the inliner for development.
By correctness I was referring to the inliner producing incorrect code on a
clean
compile. Yes it's known that it has issues when done on local development
due to
incremental compilation which is why there is a knob to disable it for
local dev.
There is already an issue[1] about reversing the default setting so that
it's
disabled by default. It probably wasn't a wise idea to have it enabled by
default locally (my bad) but the original motivation for that was because
we can't make release artifacts in CI yet, it was to reduce the chance of
creating an incorrect release artifact however as stated below this is
likely a non-issue.
> If we decide to run the inliner on CI and for releases, then this will
introduce another level of complexity to important steps of the
release pipeline building differently in different situations.
Our release process regardless of inliner is already complex enough[2] and
hence
we are already having to pass custom flags[3] when creating release
artifacts
anyways so I don't see how adding an extra single flag i.e.
pekko.inline=true to
sbt is going to materially affect the complexity either way. It's also
already
document to +clean the project for reasons aside from the inliner, i.e. the
incremental compiler even with inliner disabled doesn't produce
deterministic
bytecode which effect's reproducible builds which is something we are
trying to
move to.
[1]:
https://github.com/pjfanning/sbt-pekko-build/issues/9#issuecomment-1914496929
[2]:
https://github.com/apache/incubator-pekko-site/wiki/Pekko-Release-Process
[3]:
https://github.com/apache/incubator-pekko-site/wiki/Pekko-Release-Process#deploy-the-jars-to-apache-maven-repository-staging
On Mon, Jan 29, 2024 at 9:31 PM Johannes Rudolph <jo...@gmail.com>
wrote:
> On Thu, Jan 18, 2024 at 12:48 AM Matthew de Detrich
> <ma...@aiven.io.invalid> wrote:
> > it will automatically inline any code that it knows won't get inlined by
> the JDK
>
> It will only do that in some select few cases.
>
> > In any case, I would still opt for leaving the inliner in there. I would
> be
> > extremely
> > shocked if there are any correctness issues given that its being used for
> > at least
> > half a decade in many projects
>
> See https://github.com/apache/incubator-pekko-http/pull/456 for an
> example of the correctness issue.
> Afaik none of the many projects enable the inliner for development.
>
> > hence the "free lunch" argument and the
> > complexity is not that high (we are dealing with some extra scalac flags)
>
> I would not call it free lunch, especially since the development
> efficiency is impacted by far longer compile times and potential
> correctness issues.
> If we decide to run the inliner on CI and for releases, then this will
> introduce another level of complexity to important steps of the
> release pipeline building differently in different situations.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Johannes Rudolph <jo...@gmail.com>.
On Thu, Jan 18, 2024 at 12:48 AM Matthew de Detrich
<ma...@aiven.io.invalid> wrote:
> it will automatically inline any code that it knows won't get inlined by the JDK
It will only do that in some select few cases.
> In any case, I would still opt for leaving the inliner in there. I would be
> extremely
> shocked if there are any correctness issues given that its being used for
> at least
> half a decade in many projects
See https://github.com/apache/incubator-pekko-http/pull/456 for an
example of the correctness issue.
Afaik none of the many projects enable the inliner for development.
> hence the "free lunch" argument and the
> complexity is not that high (we are dealing with some extra scalac flags)
I would not call it free lunch, especially since the development
efficiency is impacted by far longer compile times and potential
correctness issues.
If we decide to run the inliner on CI and for releases, then this will
introduce another level of complexity to important steps of the
release pipeline building differently in different situations.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
For additional commands, e-mail: dev-help@pekko.apache.org
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> Thanks for running these, Matthew. Alas, I cannot reproduce the
difference. How often did you run to reproduce the noticeable results
yourself?
I did re-run the entire benchmark 2-3 times. There was definitely a
deviation of around +-5% but I there was a general trend in which
the inliner version was faster
> Here are a few more ideas:
> - rather clock the cpu down and keep check of the actual frequencies
while running (e.g. using perf stat), instead of turning the fans up.
There might still be thermal reservoirs of unknown size and state even
with highest fan settings, especially in laptops.
> - always run more than one fork, many benchmarks have (at least)
bimodal distributions of performance in steady state varying by a few
percent. If you know there's a multimodal distribution, make a rough
estimation how many forks you need to really make sure aren't
comparing one coin flip with the other one on the other side.
> - if the error bars overlap a lot that's a sign that you need to run
more benchmarks for validation (jmh reports a large 99.9% CI but make
sure you are confident the interval itself makes sense, 3 measurements
are really the bare minimum for those to make any sense at all)
Yes I am aware of these but since I am overseas they are being run
on an M1 laptop which is not desirable (hence why I said
beforehand take the results as gospel before). Afaik due to how M1 is
designed
there isn't a proper way to do things like freeze clocks, the best you can
do
is to make sure the thermals are as constant as possible.
At home I have a desktop setup specifically for benchmarking, i.e. frozen
clocks in bios + scheduler tweaks etc etc but I don't have access to it now.
Hence why I am pushing for an actual proper dedicated HW solution
> Also the result seems somewhat cherry-picked as e.g. most of the
LineParserBenchmarks were slightly (up to 7%) better without inlining
(speaking about your results in all of the above).
Yes I also noticed this, from quick grocking the reverse case was less
frequent but I think we are in agreement that we shouldn't make
any definite conclusions here.
> Btw. the main improvement that the inliner promises is that it can
avoid a particular kind of megamorphic call site in higher-order
function use, i.e. scala collection usage. In most performance
critical situations, this has been manually optimized before where it
turned up in profiles. All the other expensive places of megamorphic
call sites (dispatchers, stream GraphInterpreter, routing DSL) are
unfortunately not "static enough" that an AOT inliner could pick them
up with static analysis.
Definitely agreed, the kicker being the "performance critical situations".
The
Scala 2 inliner is having an impact, when enabled it will automatically
inline
any code that it knows won't get inlined by the JDK (particularly the
earlier
ones like JDK 1.8, I heard later JDK's/GraalVM do a better job of this) and
as
you can tell from the bytecode diff there were a lot of cases like this.
This is what
I meant earlier when I was saying I would be surprised if there was
a significant difference. I would presume that the Akka already did their
best to
optimize any actual performance critical hotspots.
In any case, I would still opt for leaving the inliner in there. I would be
extremely
shocked if there are any correctness issues given that its being used for
at least
half a decade in many projects hence the "free lunch" argument and the
complexity is not that high (we are dealing with some extra scalac flags)
and
we are almost finished with enabling it. Also in in terms of timing now is
the
best time, giving Pekko 1.1.x is on the horizon, the intent of doing
these changes
now is so that they won't be done later at more inopportune times
(I am predicting Pekko 1.1.x to last a while).
One obvious advantage the inliner does have is for newly contributed code,
it
can take care of some hotspots for us which given how we are trying to build
up contributors now is a nice bonus, not every contributor to Pekko is knee
deep into JVM performance and we don't have the setup/labor now to profile
problematic cases like Lightbend/Akka did/has.
If there are any critical issues exposed from the inliner I can definitely
look into
reverting it, but I already did a lot of preparation work plan for this so
I am not
expecting anything.
On Thu, Jan 18, 2024 at 2:17 AM Johannes Rudolph <jo...@gmail.com>
wrote:
> On Wed, Jan 17, 2024 at 2:07 AM Matthew de Detrich
> <ma...@aiven.io.invalid> wrote:
> > As you can see from the results there are some noticeable improvements
> > (i.e. 5-10% in some cases) however I wouldn't take these results as
> > complete gospel as I had to do the benchmarks on my M1 laptop (I had it
> in
> > power plus used TGPro to put fans on max blast to reduce any variability,
> > unfortunately I am currently overseas so I don't have a dedicated machine
> > to test on).
>
> Thanks for running these, Matthew. Alas, I cannot reproduce the
> difference. How often did you run to reproduce the noticeable results
> yourself?
>
> Here are a few more ideas:
>
> - rather clock the cpu down and keep check of the actual frequencies
> while running (e.g. using perf stat), instead of turning the fans up.
> There might still be thermal reservoirs of unknown size and state even
> with highest fan settings, especially in laptops.
> - always run more than one fork, many benchmarks have (at least)
> bimodal distributions of performance in steady state varying by a few
> percent. If you know there's a multimodal distribution, make a rough
> estimation how many forks you need to really make sure aren't
> comparing one coin flip with the other one on the other side.
> - if the error bars overlap a lot that's a sign that you need to run
> more benchmarks for validation (jmh reports a large 99.9% CI but make
> sure you are confident the interval itself makes sense, 3 measurements
> are really the bare minimum for those to make any sense at all)
>
> Many of these issues are also taken care of by creating a (e.g.
> nightly) long-running benchmark series where the random fluctuations
> become quite apparent over days.
>
> In general, for complex benchmarks like in pekko-http, I like to use
> these rough guidelines for evaluating benchmark evidence:
>
> * < 5% difference needs exceptional statistical evidence and a
> reasonable explanation for the behavior (e.g. you tried to optimize
> something before and the improvements are exactly in the area that you
> expected)
> * 5-10% difference needs very good statistical evidence and/or
> explanations for the improvements
> * ...
> * > 10-15% if consistently better in multiple runs and environment,
> likely an improvement
>
> (When benchmarking single methods you might relax the judgement,
> though then the measured performance might not materialize in more
> realistic scenarios)
>
> The StreamedServerProcessing result seems somewhat internally
> inconsistent since the same "chunked" configuration with different
> chunk sizes shows somewhat different behavior which is possible but
> maybe not super likely?
>
> Also the result seems somewhat cherry-picked as e.g. most of the
> LineParserBenchmarks were slightly (up to 7%) better without inlining
> (speaking about your results in all of the above).
>
> Here are my quick results (also very weak evidence):
> https://gist.github.com/jrudolph/bc97146dedf0290d059e5e44939fbdc0
>
> Btw. the main improvement that the inliner promises is that it can
> avoid a particular kind of megamorphic call site in higher-order
> function use, i.e. scala collection usage. In most performance
> critical situations, this has been manually optimized before where it
> turned up in profiles. All the other expensive places of megamorphic
> call sites (dispatchers, stream GraphInterpreter, routing DSL) are
> unfortunately not "static enough" that an AOT inliner could pick them
> up with static analysis.
>
> Johannes
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
> For additional commands, e-mail: dev-help@pekko.apache.org
>
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Johannes Rudolph <jo...@gmail.com>.
On Wed, Jan 17, 2024 at 2:07 AM Matthew de Detrich
<ma...@aiven.io.invalid> wrote:
> As you can see from the results there are some noticeable improvements
> (i.e. 5-10% in some cases) however I wouldn't take these results as
> complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
> power plus used TGPro to put fans on max blast to reduce any variability,
> unfortunately I am currently overseas so I don't have a dedicated machine
> to test on).
Thanks for running these, Matthew. Alas, I cannot reproduce the
difference. How often did you run to reproduce the noticeable results
yourself?
Here are a few more ideas:
- rather clock the cpu down and keep check of the actual frequencies
while running (e.g. using perf stat), instead of turning the fans up.
There might still be thermal reservoirs of unknown size and state even
with highest fan settings, especially in laptops.
- always run more than one fork, many benchmarks have (at least)
bimodal distributions of performance in steady state varying by a few
percent. If you know there's a multimodal distribution, make a rough
estimation how many forks you need to really make sure aren't
comparing one coin flip with the other one on the other side.
- if the error bars overlap a lot that's a sign that you need to run
more benchmarks for validation (jmh reports a large 99.9% CI but make
sure you are confident the interval itself makes sense, 3 measurements
are really the bare minimum for those to make any sense at all)
Many of these issues are also taken care of by creating a (e.g.
nightly) long-running benchmark series where the random fluctuations
become quite apparent over days.
In general, for complex benchmarks like in pekko-http, I like to use
these rough guidelines for evaluating benchmark evidence:
* < 5% difference needs exceptional statistical evidence and a
reasonable explanation for the behavior (e.g. you tried to optimize
something before and the improvements are exactly in the area that you
expected)
* 5-10% difference needs very good statistical evidence and/or
explanations for the improvements
* ...
* > 10-15% if consistently better in multiple runs and environment,
likely an improvement
(When benchmarking single methods you might relax the judgement,
though then the measured performance might not materialize in more
realistic scenarios)
The StreamedServerProcessing result seems somewhat internally
inconsistent since the same "chunked" configuration with different
chunk sizes shows somewhat different behavior which is possible but
maybe not super likely?
Also the result seems somewhat cherry-picked as e.g. most of the
LineParserBenchmarks were slightly (up to 7%) better without inlining
(speaking about your results in all of the above).
Here are my quick results (also very weak evidence):
https://gist.github.com/jrudolph/bc97146dedf0290d059e5e44939fbdc0
Btw. the main improvement that the inliner promises is that it can
avoid a particular kind of megamorphic call site in higher-order
function use, i.e. scala collection usage. In most performance
critical situations, this has been manually optimized before where it
turned up in profiles. All the other expensive places of megamorphic
call sites (dispatchers, stream GraphInterpreter, routing DSL) are
unfortunately not "static enough" that an AOT inliner could pick them
up with static analysis.
Johannes
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pekko.apache.org
For additional commands, e-mail: dev-help@pekko.apache.org
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> Thanks for sharing, but for the test server, I think we can use
https://www.gitpod.io/discover/opensource
When it comes to benchmarking and also nightly tests, a dedicated machine
is much
more reliable because it doesn't have the noisy neighbour problem, i.e. its
more deterministic since it doesn't have to share resources with other
virtual machines.
On Thu, Jan 18, 2024 at 12:07 AM kerr <he...@gmail.com> wrote:
> Thanks for sharing, but for the test server, I think we can use
> https://www.gitpod.io/discover/opensource
>
> Which I was using it for testing the netty 4 migration.
>
> And I think it would be nice if you have time to test the JDK21 too.
>
> 何品
>
>
> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
> 09:35写道：
>
> > And if anyone is interested in the bytecode diffs they can look at
> >
> >
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
> >
> > On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
> > matthew.dedetrich@aiven.io> wrote:
> >
> > > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
> > since
> > > this
> > > is where the Scala2 inliner makes different impacts, latest version of
> > > JDK's should
> > > do a better job of inlining problematic bytecode generated by Scala
> but a
> > > significant
> > > part of our user base is still running JDK8.
> > >
> > > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> > > matthew.dedetrich@aiven.io> wrote:
> > >
> > >> Since the inliner has been merged into both Pekko and Pekko Http I
> > >> decided to post some results for pekko-http's jmh bench when all of
> the
> > >> inlining settings are applied (left is without inliner, right is with
> > >> inliner)
> > >>
> > >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> > >> Against Pekko 1.0.0-M1 (current main):
> > >> https://www.diffchecker.com/XhyEC556/
> > >>
> > >> As you can see from the results there are some noticeable improvements
> > >> (i.e. 5-10% in some cases) however I wouldn't take these results as
> > >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
> it
> > in
> > >> power plus used TGPro to put fans on max blast to reduce any
> > variability,
> > >> unfortunately I am currently overseas so I don't have a dedicated
> > machine
> > >> to test on).
> > >>
> > >> This testing was a result of valid concerns raised by Johannes[1].
> > >> Another valid point being raised is about being diligent when it comes
> > to
> > >> performance/benchmarks which is what dedicated hardware for
> benchmarking
> > >> was meant to solve. PJ Fanning has set up an open collective for
> > Pekko[2]
> > >> and there is already a github discussion[3] about it.
> > >>
> > >> So maybe now is a good time to start talking about funding for the
> Pekko
> > >> project in general as well as being a call to action to any heavy
> users
> > of
> > >> Pekko that feel like donating? The initial plan was to use the funds
> to
> > >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> > >> dedicated bare metal machines costing 50-100 euros a month, with one
> > >> machine being used for nightly builds and another machine dedicated to
> > >> benchmarking.
> > >>
> > >> We can also discuss what the additional funds can be used for (i.e.
> > >> sponsoring core maintainers, talks/conferences once Pekko gets big
> > enough
> > >> etc etc?) although that might be best for another thread.
> > >>
> > >> Thoughts?
> > >>
> > >> [1]:
> > >>
> >
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> > >> [2]: https://opencollective.com/friends-of-apache-pekko
> > >> [3]: https://github.com/apache/incubator-pekko/discussions/144
> > >> --
> > >>
> > >> Matthew de Detrich
> > >>
> > >> *Aiven Deutschland GmbH*
> > >>
> > >> Immanuelkirchstraße 26, 10405 Berlin
> > >>
> > >> Alexanderufer 3-7, 10117 Berlin
> > >>
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>
> > >> *m:* +491603708037
> > >>
> > >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> > >>
> > >
> > >
> > > --
> > >
> > > Matthew de Detrich
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491603708037
> > >
> > > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> > >
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
> Thanks for sharing, but for the test server, I think we can use
https://www.gitpod.io/discover/opensource
When it comes to benchmarking and also nightly tests, a dedicated machine
is much
more reliable because it doesn't have the noisy neighbour problem, i.e. its
more deterministic since it doesn't have to share resources with other
virtual machines.
On Thu, Jan 18, 2024 at 12:07 AM kerr <he...@gmail.com> wrote:
> Thanks for sharing, but for the test server, I think we can use
> https://www.gitpod.io/discover/opensource
>
> Which I was using it for testing the netty 4 migration.
>
> And I think it would be nice if you have time to test the JDK21 too.
>
> 何品
>
>
> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
> 09:35写道：
>
> > And if anyone is interested in the bytecode diffs they can look at
> >
> >
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
> >
> > On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
> > matthew.dedetrich@aiven.io> wrote:
> >
> > > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
> > since
> > > this
> > > is where the Scala2 inliner makes different impacts, latest version of
> > > JDK's should
> > > do a better job of inlining problematic bytecode generated by Scala
> but a
> > > significant
> > > part of our user base is still running JDK8.
> > >
> > > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> > > matthew.dedetrich@aiven.io> wrote:
> > >
> > >> Since the inliner has been merged into both Pekko and Pekko Http I
> > >> decided to post some results for pekko-http's jmh bench when all of
> the
> > >> inlining settings are applied (left is without inliner, right is with
> > >> inliner)
> > >>
> > >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> > >> Against Pekko 1.0.0-M1 (current main):
> > >> https://www.diffchecker.com/XhyEC556/
> > >>
> > >> As you can see from the results there are some noticeable improvements
> > >> (i.e. 5-10% in some cases) however I wouldn't take these results as
> > >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
> it
> > in
> > >> power plus used TGPro to put fans on max blast to reduce any
> > variability,
> > >> unfortunately I am currently overseas so I don't have a dedicated
> > machine
> > >> to test on).
> > >>
> > >> This testing was a result of valid concerns raised by Johannes[1].
> > >> Another valid point being raised is about being diligent when it comes
> > to
> > >> performance/benchmarks which is what dedicated hardware for
> benchmarking
> > >> was meant to solve. PJ Fanning has set up an open collective for
> > Pekko[2]
> > >> and there is already a github discussion[3] about it.
> > >>
> > >> So maybe now is a good time to start talking about funding for the
> Pekko
> > >> project in general as well as being a call to action to any heavy
> users
> > of
> > >> Pekko that feel like donating? The initial plan was to use the funds
> to
> > >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> > >> dedicated bare metal machines costing 50-100 euros a month, with one
> > >> machine being used for nightly builds and another machine dedicated to
> > >> benchmarking.
> > >>
> > >> We can also discuss what the additional funds can be used for (i.e.
> > >> sponsoring core maintainers, talks/conferences once Pekko gets big
> > enough
> > >> etc etc?) although that might be best for another thread.
> > >>
> > >> Thoughts?
> > >>
> > >> [1]:
> > >>
> >
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> > >> [2]: https://opencollective.com/friends-of-apache-pekko
> > >> [3]: https://github.com/apache/incubator-pekko/discussions/144
> > >> --
> > >>
> > >> Matthew de Detrich
> > >>
> > >> *Aiven Deutschland GmbH*
> > >>
> > >> Immanuelkirchstraße 26, 10405 Berlin
> > >>
> > >> Alexanderufer 3-7, 10117 Berlin
> > >>
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>
> > >> *m:* +491603708037
> > >>
> > >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> > >>
> > >
> > >
> > > --
> > >
> > > Matthew de Detrich
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491603708037
> > >
> > > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> > >
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
Additionally, congratulations on all your hard work paying off. another
question is how about the flamegraph？I think you can capture it for
difference too.
eg: https://github.com/netty/netty/pull/13759 he attached a graph.
何品
kerr <he...@gmail.com> 于2024年1月17日周三 21:08写道：
> As we have submitted the pekko-grpc to
> https://github.com/LesnyRumcajs/grpc_bench, I think we will see some
> performance boost there too.
>
> 何品
>
>
> kerr <he...@gmail.com> 于2024年1月17日周三 21:07写道：
>
>> Thanks for sharing, but for the test server, I think we can use
>> https://www.gitpod.io/discover/opensource
>>
>> Which I was using it for testing the netty 4 migration.
>>
>> And I think it would be nice if you have time to test the JDK21 too.
>>
>> 何品
>>
>>
>> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
>> 09:35写道：
>>
>>> And if anyone is interested in the bytecode diffs they can look at
>>>
>>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>>>
>>> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
>>> matthew.dedetrich@aiven.io> wrote:
>>>
>>> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
>>> since
>>> > this
>>> > is where the Scala2 inliner makes different impacts, latest version of
>>> > JDK's should
>>> > do a better job of inlining problematic bytecode generated by Scala
>>> but a
>>> > significant
>>> > part of our user base is still running JDK8.
>>> >
>>> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
>>> > matthew.dedetrich@aiven.io> wrote:
>>> >
>>> >> Since the inliner has been merged into both Pekko and Pekko Http I
>>> >> decided to post some results for pekko-http's jmh bench when all of
>>> the
>>> >> inlining settings are applied (left is without inliner, right is with
>>> >> inliner)
>>> >>
>>> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>>> >> Against Pekko 1.0.0-M1 (current main):
>>> >> https://www.diffchecker.com/XhyEC556/
>>> >>
>>> >> As you can see from the results there are some noticeable improvements
>>> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
>>> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
>>> it in
>>> >> power plus used TGPro to put fans on max blast to reduce any
>>> variability,
>>> >> unfortunately I am currently overseas so I don't have a dedicated
>>> machine
>>> >> to test on).
>>> >>
>>> >> This testing was a result of valid concerns raised by Johannes[1].
>>> >> Another valid point being raised is about being diligent when it
>>> comes to
>>> >> performance/benchmarks which is what dedicated hardware for
>>> benchmarking
>>> >> was meant to solve. PJ Fanning has set up an open collective for
>>> Pekko[2]
>>> >> and there is already a github discussion[3] about it.
>>> >>
>>> >> So maybe now is a good time to start talking about funding for the
>>> Pekko
>>> >> project in general as well as being a call to action to any heavy
>>> users of
>>> >> Pekko that feel like donating? The initial plan was to use the funds
>>> to
>>> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>>> >> dedicated bare metal machines costing 50-100 euros a month, with one
>>> >> machine being used for nightly builds and another machine dedicated to
>>> >> benchmarking.
>>> >>
>>> >> We can also discuss what the additional funds can be used for (i.e.
>>> >> sponsoring core maintainers, talks/conferences once Pekko gets big
>>> enough
>>> >> etc etc?) although that might be best for another thread.
>>> >>
>>> >> Thoughts?
>>> >>
>>> >> [1]:
>>> >>
>>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>>> >> [2]: https://opencollective.com/friends-of-apache-pekko
>>> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
>>> >> --
>>> >>
>>> >> Matthew de Detrich
>>> >>
>>> >> *Aiven Deutschland GmbH*
>>> >>
>>> >> Immanuelkirchstraße 26, 10405 Berlin
>>> >>
>>> >> Alexanderufer 3-7, 10117 Berlin
>>> >>
>>> >> Amtsgericht Charlottenburg, HRB 209739 B
>>> >>
>>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >>
>>> >> *m:* +491603708037
>>> >>
>>> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>> >>
>>> >
>>> >
>>> > --
>>> >
>>> > Matthew de Detrich
>>> >
>>> > *Aiven Deutschland GmbH*
>>> >
>>> > Immanuelkirchstraße 26, 10405 Berlin
>>> >
>>> > Alexanderufer 3-7, 10117 Berlin
>>> >
>>> > Amtsgericht Charlottenburg, HRB 209739 B
>>> >
>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >
>>> > *m:* +491603708037
>>> >
>>> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>> >
>>>
>>>
>>> --
>>>
>>> Matthew de Detrich
>>>
>>> *Aiven Deutschland GmbH*
>>>
>>> Immanuelkirchstraße 26, 10405 Berlin
>>>
>>> Alexanderufer 3-7, 10117 Berlin
>>>
>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>
>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>>
>>> *m:* +491603708037
>>>
>>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>>
>>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
Additionally, congratulations on all your hard work paying off. another
question is how about the flamegraph？I think you can capture it for
difference too.
eg: https://github.com/netty/netty/pull/13759 he attached a graph.
何品
kerr <he...@gmail.com> 于2024年1月17日周三 21:08写道：
> As we have submitted the pekko-grpc to
> https://github.com/LesnyRumcajs/grpc_bench, I think we will see some
> performance boost there too.
>
> 何品
>
>
> kerr <he...@gmail.com> 于2024年1月17日周三 21:07写道：
>
>> Thanks for sharing, but for the test server, I think we can use
>> https://www.gitpod.io/discover/opensource
>>
>> Which I was using it for testing the netty 4 migration.
>>
>> And I think it would be nice if you have time to test the JDK21 too.
>>
>> 何品
>>
>>
>> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
>> 09:35写道：
>>
>>> And if anyone is interested in the bytecode diffs they can look at
>>>
>>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>>>
>>> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
>>> matthew.dedetrich@aiven.io> wrote:
>>>
>>> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
>>> since
>>> > this
>>> > is where the Scala2 inliner makes different impacts, latest version of
>>> > JDK's should
>>> > do a better job of inlining problematic bytecode generated by Scala
>>> but a
>>> > significant
>>> > part of our user base is still running JDK8.
>>> >
>>> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
>>> > matthew.dedetrich@aiven.io> wrote:
>>> >
>>> >> Since the inliner has been merged into both Pekko and Pekko Http I
>>> >> decided to post some results for pekko-http's jmh bench when all of
>>> the
>>> >> inlining settings are applied (left is without inliner, right is with
>>> >> inliner)
>>> >>
>>> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>>> >> Against Pekko 1.0.0-M1 (current main):
>>> >> https://www.diffchecker.com/XhyEC556/
>>> >>
>>> >> As you can see from the results there are some noticeable improvements
>>> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
>>> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
>>> it in
>>> >> power plus used TGPro to put fans on max blast to reduce any
>>> variability,
>>> >> unfortunately I am currently overseas so I don't have a dedicated
>>> machine
>>> >> to test on).
>>> >>
>>> >> This testing was a result of valid concerns raised by Johannes[1].
>>> >> Another valid point being raised is about being diligent when it
>>> comes to
>>> >> performance/benchmarks which is what dedicated hardware for
>>> benchmarking
>>> >> was meant to solve. PJ Fanning has set up an open collective for
>>> Pekko[2]
>>> >> and there is already a github discussion[3] about it.
>>> >>
>>> >> So maybe now is a good time to start talking about funding for the
>>> Pekko
>>> >> project in general as well as being a call to action to any heavy
>>> users of
>>> >> Pekko that feel like donating? The initial plan was to use the funds
>>> to
>>> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>>> >> dedicated bare metal machines costing 50-100 euros a month, with one
>>> >> machine being used for nightly builds and another machine dedicated to
>>> >> benchmarking.
>>> >>
>>> >> We can also discuss what the additional funds can be used for (i.e.
>>> >> sponsoring core maintainers, talks/conferences once Pekko gets big
>>> enough
>>> >> etc etc?) although that might be best for another thread.
>>> >>
>>> >> Thoughts?
>>> >>
>>> >> [1]:
>>> >>
>>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>>> >> [2]: https://opencollective.com/friends-of-apache-pekko
>>> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
>>> >> --
>>> >>
>>> >> Matthew de Detrich
>>> >>
>>> >> *Aiven Deutschland GmbH*
>>> >>
>>> >> Immanuelkirchstraße 26, 10405 Berlin
>>> >>
>>> >> Alexanderufer 3-7, 10117 Berlin
>>> >>
>>> >> Amtsgericht Charlottenburg, HRB 209739 B
>>> >>
>>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >>
>>> >> *m:* +491603708037
>>> >>
>>> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>> >>
>>> >
>>> >
>>> > --
>>> >
>>> > Matthew de Detrich
>>> >
>>> > *Aiven Deutschland GmbH*
>>> >
>>> > Immanuelkirchstraße 26, 10405 Berlin
>>> >
>>> > Alexanderufer 3-7, 10117 Berlin
>>> >
>>> > Amtsgericht Charlottenburg, HRB 209739 B
>>> >
>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >
>>> > *m:* +491603708037
>>> >
>>> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>> >
>>>
>>>
>>> --
>>>
>>> Matthew de Detrich
>>>
>>> *Aiven Deutschland GmbH*
>>>
>>> Immanuelkirchstraße 26, 10405 Berlin
>>>
>>> Alexanderufer 3-7, 10117 Berlin
>>>
>>> Amtsgericht Charlottenburg, HRB 209739 B
>>>
>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>>
>>> *m:* +491603708037
>>>
>>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>>
>>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
As we have submitted the pekko-grpc to
https://github.com/LesnyRumcajs/grpc_bench, I think we will see some
performance boost there too.
何品
kerr <he...@gmail.com> 于2024年1月17日周三 21:07写道：
> Thanks for sharing, but for the test server, I think we can use
> https://www.gitpod.io/discover/opensource
>
> Which I was using it for testing the netty 4 migration.
>
> And I think it would be nice if you have time to test the JDK21 too.
>
> 何品
>
>
> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
> 09:35写道：
>
>> And if anyone is interested in the bytecode diffs they can look at
>>
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>>
>> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
>> matthew.dedetrich@aiven.io> wrote:
>>
>> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
>> since
>> > this
>> > is where the Scala2 inliner makes different impacts, latest version of
>> > JDK's should
>> > do a better job of inlining problematic bytecode generated by Scala but
>> a
>> > significant
>> > part of our user base is still running JDK8.
>> >
>> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
>> > matthew.dedetrich@aiven.io> wrote:
>> >
>> >> Since the inliner has been merged into both Pekko and Pekko Http I
>> >> decided to post some results for pekko-http's jmh bench when all of the
>> >> inlining settings are applied (left is without inliner, right is with
>> >> inliner)
>> >>
>> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>> >> Against Pekko 1.0.0-M1 (current main):
>> >> https://www.diffchecker.com/XhyEC556/
>> >>
>> >> As you can see from the results there are some noticeable improvements
>> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
>> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
>> it in
>> >> power plus used TGPro to put fans on max blast to reduce any
>> variability,
>> >> unfortunately I am currently overseas so I don't have a dedicated
>> machine
>> >> to test on).
>> >>
>> >> This testing was a result of valid concerns raised by Johannes[1].
>> >> Another valid point being raised is about being diligent when it comes
>> to
>> >> performance/benchmarks which is what dedicated hardware for
>> benchmarking
>> >> was meant to solve. PJ Fanning has set up an open collective for
>> Pekko[2]
>> >> and there is already a github discussion[3] about it.
>> >>
>> >> So maybe now is a good time to start talking about funding for the
>> Pekko
>> >> project in general as well as being a call to action to any heavy
>> users of
>> >> Pekko that feel like donating? The initial plan was to use the funds to
>> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>> >> dedicated bare metal machines costing 50-100 euros a month, with one
>> >> machine being used for nightly builds and another machine dedicated to
>> >> benchmarking.
>> >>
>> >> We can also discuss what the additional funds can be used for (i.e.
>> >> sponsoring core maintainers, talks/conferences once Pekko gets big
>> enough
>> >> etc etc?) although that might be best for another thread.
>> >>
>> >> Thoughts?
>> >>
>> >> [1]:
>> >>
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>> >> [2]: https://opencollective.com/friends-of-apache-pekko
>> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
>> >> --
>> >>
>> >> Matthew de Detrich
>> >>
>> >> *Aiven Deutschland GmbH*
>> >>
>> >> Immanuelkirchstraße 26, 10405 Berlin
>> >>
>> >> Alexanderufer 3-7, 10117 Berlin
>> >>
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >>
>> >> *m:* +491603708037
>> >>
>> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>> >>
>> >
>> >
>> > --
>> >
>> > Matthew de Detrich
>> >
>> > *Aiven Deutschland GmbH*
>> >
>> > Immanuelkirchstraße 26, 10405 Berlin
>> >
>> > Alexanderufer 3-7, 10117 Berlin
>> >
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >
>> > *m:* +491603708037
>> >
>> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>> >
>>
>>
>> --
>>
>> Matthew de Detrich
>>
>> *Aiven Deutschland GmbH*
>>
>> Immanuelkirchstraße 26, 10405 Berlin
>>
>> Alexanderufer 3-7, 10117 Berlin
>>
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>
>> *m:* +491603708037
>>
>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
As we have submitted the pekko-grpc to
https://github.com/LesnyRumcajs/grpc_bench, I think we will see some
performance boost there too.
何品
kerr <he...@gmail.com> 于2024年1月17日周三 21:07写道：
> Thanks for sharing, but for the test server, I think we can use
> https://www.gitpod.io/discover/opensource
>
> Which I was using it for testing the netty 4 migration.
>
> And I think it would be nice if you have time to test the JDK21 too.
>
> 何品
>
>
> Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
> 09:35写道：
>
>> And if anyone is interested in the bytecode diffs they can look at
>>
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>>
>> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
>> matthew.dedetrich@aiven.io> wrote:
>>
>> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
>> since
>> > this
>> > is where the Scala2 inliner makes different impacts, latest version of
>> > JDK's should
>> > do a better job of inlining problematic bytecode generated by Scala but
>> a
>> > significant
>> > part of our user base is still running JDK8.
>> >
>> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
>> > matthew.dedetrich@aiven.io> wrote:
>> >
>> >> Since the inliner has been merged into both Pekko and Pekko Http I
>> >> decided to post some results for pekko-http's jmh bench when all of the
>> >> inlining settings are applied (left is without inliner, right is with
>> >> inliner)
>> >>
>> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>> >> Against Pekko 1.0.0-M1 (current main):
>> >> https://www.diffchecker.com/XhyEC556/
>> >>
>> >> As you can see from the results there are some noticeable improvements
>> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
>> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had
>> it in
>> >> power plus used TGPro to put fans on max blast to reduce any
>> variability,
>> >> unfortunately I am currently overseas so I don't have a dedicated
>> machine
>> >> to test on).
>> >>
>> >> This testing was a result of valid concerns raised by Johannes[1].
>> >> Another valid point being raised is about being diligent when it comes
>> to
>> >> performance/benchmarks which is what dedicated hardware for
>> benchmarking
>> >> was meant to solve. PJ Fanning has set up an open collective for
>> Pekko[2]
>> >> and there is already a github discussion[3] about it.
>> >>
>> >> So maybe now is a good time to start talking about funding for the
>> Pekko
>> >> project in general as well as being a call to action to any heavy
>> users of
>> >> Pekko that feel like donating? The initial plan was to use the funds to
>> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>> >> dedicated bare metal machines costing 50-100 euros a month, with one
>> >> machine being used for nightly builds and another machine dedicated to
>> >> benchmarking.
>> >>
>> >> We can also discuss what the additional funds can be used for (i.e.
>> >> sponsoring core maintainers, talks/conferences once Pekko gets big
>> enough
>> >> etc etc?) although that might be best for another thread.
>> >>
>> >> Thoughts?
>> >>
>> >> [1]:
>> >>
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>> >> [2]: https://opencollective.com/friends-of-apache-pekko
>> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
>> >> --
>> >>
>> >> Matthew de Detrich
>> >>
>> >> *Aiven Deutschland GmbH*
>> >>
>> >> Immanuelkirchstraße 26, 10405 Berlin
>> >>
>> >> Alexanderufer 3-7, 10117 Berlin
>> >>
>> >> Amtsgericht Charlottenburg, HRB 209739 B
>> >>
>> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >>
>> >> *m:* +491603708037
>> >>
>> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>> >>
>> >
>> >
>> > --
>> >
>> > Matthew de Detrich
>> >
>> > *Aiven Deutschland GmbH*
>> >
>> > Immanuelkirchstraße 26, 10405 Berlin
>> >
>> > Alexanderufer 3-7, 10117 Berlin
>> >
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> >
>> > *m:* +491603708037
>> >
>> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>> >
>>
>>
>> --
>>
>> Matthew de Detrich
>>
>> *Aiven Deutschland GmbH*
>>
>> Immanuelkirchstraße 26, 10405 Berlin
>>
>> Alexanderufer 3-7, 10117 Berlin
>>
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>
>> *m:* +491603708037
>>
>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
Thanks for sharing, but for the test server, I think we can use
https://www.gitpod.io/discover/opensource
Which I was using it for testing the netty 4 migration.
And I think it would be nice if you have time to test the JDK21 too.
何品
Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
09:35写道：
> And if anyone is interested in the bytecode diffs they can look at
>
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>
> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
> matthew.dedetrich@aiven.io> wrote:
>
> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
> since
> > this
> > is where the Scala2 inliner makes different impacts, latest version of
> > JDK's should
> > do a better job of inlining problematic bytecode generated by Scala but a
> > significant
> > part of our user base is still running JDK8.
> >
> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> > matthew.dedetrich@aiven.io> wrote:
> >
> >> Since the inliner has been merged into both Pekko and Pekko Http I
> >> decided to post some results for pekko-http's jmh bench when all of the
> >> inlining settings are applied (left is without inliner, right is with
> >> inliner)
> >>
> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> >> Against Pekko 1.0.0-M1 (current main):
> >> https://www.diffchecker.com/XhyEC556/
> >>
> >> As you can see from the results there are some noticeable improvements
> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had it
> in
> >> power plus used TGPro to put fans on max blast to reduce any
> variability,
> >> unfortunately I am currently overseas so I don't have a dedicated
> machine
> >> to test on).
> >>
> >> This testing was a result of valid concerns raised by Johannes[1].
> >> Another valid point being raised is about being diligent when it comes
> to
> >> performance/benchmarks which is what dedicated hardware for benchmarking
> >> was meant to solve. PJ Fanning has set up an open collective for
> Pekko[2]
> >> and there is already a github discussion[3] about it.
> >>
> >> So maybe now is a good time to start talking about funding for the Pekko
> >> project in general as well as being a call to action to any heavy users
> of
> >> Pekko that feel like donating? The initial plan was to use the funds to
> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> >> dedicated bare metal machines costing 50-100 euros a month, with one
> >> machine being used for nightly builds and another machine dedicated to
> >> benchmarking.
> >>
> >> We can also discuss what the additional funds can be used for (i.e.
> >> sponsoring core maintainers, talks/conferences once Pekko gets big
> enough
> >> etc etc?) although that might be best for another thread.
> >>
> >> Thoughts?
> >>
> >> [1]:
> >>
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> >> [2]: https://opencollective.com/friends-of-apache-pekko
> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
> >> --
> >>
> >> Matthew de Detrich
> >>
> >> *Aiven Deutschland GmbH*
> >>
> >> Immanuelkirchstraße 26, 10405 Berlin
> >>
> >> Alexanderufer 3-7, 10117 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> *m:* +491603708037
> >>
> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >>
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by kerr <he...@gmail.com>.
Thanks for sharing, but for the test server, I think we can use
https://www.gitpod.io/discover/opensource
Which I was using it for testing the netty 4 migration.
And I think it would be nice if you have time to test the JDK21 too.
何品
Matthew de Detrich <ma...@aiven.io.invalid> 于2024年1月17日周三
09:35写道：
> And if anyone is interested in the bytecode diffs they can look at
>
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
>
> On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
> matthew.dedetrich@aiven.io> wrote:
>
> > Oh and I forgot to mention that I was using JDK 8 for the benchmarks
> since
> > this
> > is where the Scala2 inliner makes different impacts, latest version of
> > JDK's should
> > do a better job of inlining problematic bytecode generated by Scala but a
> > significant
> > part of our user base is still running JDK8.
> >
> > On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> > matthew.dedetrich@aiven.io> wrote:
> >
> >> Since the inliner has been merged into both Pekko and Pekko Http I
> >> decided to post some results for pekko-http's jmh bench when all of the
> >> inlining settings are applied (left is without inliner, right is with
> >> inliner)
> >>
> >> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> >> Against Pekko 1.0.0-M1 (current main):
> >> https://www.diffchecker.com/XhyEC556/
> >>
> >> As you can see from the results there are some noticeable improvements
> >> (i.e. 5-10% in some cases) however I wouldn't take these results as
> >> complete gospel as I had to do the benchmarks on my M1 laptop (I had it
> in
> >> power plus used TGPro to put fans on max blast to reduce any
> variability,
> >> unfortunately I am currently overseas so I don't have a dedicated
> machine
> >> to test on).
> >>
> >> This testing was a result of valid concerns raised by Johannes[1].
> >> Another valid point being raised is about being diligent when it comes
> to
> >> performance/benchmarks which is what dedicated hardware for benchmarking
> >> was meant to solve. PJ Fanning has set up an open collective for
> Pekko[2]
> >> and there is already a github discussion[3] about it.
> >>
> >> So maybe now is a good time to start talking about funding for the Pekko
> >> project in general as well as being a call to action to any heavy users
> of
> >> Pekko that feel like donating? The initial plan was to use the funds to
> >> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> >> dedicated bare metal machines costing 50-100 euros a month, with one
> >> machine being used for nightly builds and another machine dedicated to
> >> benchmarking.
> >>
> >> We can also discuss what the additional funds can be used for (i.e.
> >> sponsoring core maintainers, talks/conferences once Pekko gets big
> enough
> >> etc etc?) although that might be best for another thread.
> >>
> >> Thoughts?
> >>
> >> [1]:
> >>
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> >> [2]: https://opencollective.com/friends-of-apache-pekko
> >> [3]: https://github.com/apache/incubator-pekko/discussions/144
> >> --
> >>
> >> Matthew de Detrich
> >>
> >> *Aiven Deutschland GmbH*
> >>
> >> Immanuelkirchstraße 26, 10405 Berlin
> >>
> >> Alexanderufer 3-7, 10117 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> *m:* +491603708037
> >>
> >> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >>
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetrich@aiven.io
> >
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
And if anyone is interested in the bytecode diffs they can look at
https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
matthew.dedetrich@aiven.io> wrote:
> Oh and I forgot to mention that I was using JDK 8 for the benchmarks since
> this
> is where the Scala2 inliner makes different impacts, latest version of
> JDK's should
> do a better job of inlining problematic bytecode generated by Scala but a
> significant
> part of our user base is still running JDK8.
>
> On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> matthew.dedetrich@aiven.io> wrote:
>
>> Since the inliner has been merged into both Pekko and Pekko Http I
>> decided to post some results for pekko-http's jmh bench when all of the
>> inlining settings are applied (left is without inliner, right is with
>> inliner)
>>
>> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>> Against Pekko 1.0.0-M1 (current main):
>> https://www.diffchecker.com/XhyEC556/
>>
>> As you can see from the results there are some noticeable improvements
>> (i.e. 5-10% in some cases) however I wouldn't take these results as
>> complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
>> power plus used TGPro to put fans on max blast to reduce any variability,
>> unfortunately I am currently overseas so I don't have a dedicated machine
>> to test on).
>>
>> This testing was a result of valid concerns raised by Johannes[1].
>> Another valid point being raised is about being diligent when it comes to
>> performance/benchmarks which is what dedicated hardware for benchmarking
>> was meant to solve. PJ Fanning has set up an open collective for Pekko[2]
>> and there is already a github discussion[3] about it.
>>
>> So maybe now is a good time to start talking about funding for the Pekko
>> project in general as well as being a call to action to any heavy users of
>> Pekko that feel like donating? The initial plan was to use the funds to
>> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>> dedicated bare metal machines costing 50-100 euros a month, with one
>> machine being used for nightly builds and another machine dedicated to
>> benchmarking.
>>
>> We can also discuss what the additional funds can be used for (i.e.
>> sponsoring core maintainers, talks/conferences once Pekko gets big enough
>> etc etc?) although that might be best for another thread.
>>
>> Thoughts?
>>
>> [1]:
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>> [2]: https://opencollective.com/friends-of-apache-pekko
>> [3]: https://github.com/apache/incubator-pekko/discussions/144
>> --
>>
>> Matthew de Detrich
>>
>> *Aiven Deutschland GmbH*
>>
>> Immanuelkirchstraße 26, 10405 Berlin
>>
>> Alexanderufer 3-7, 10117 Berlin
>>
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>
>> *m:* +491603708037
>>
>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
And if anyone is interested in the bytecode diffs they can look at
https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1894791312
On Wed, Jan 17, 2024 at 12:09 PM Matthew de Detrich <
matthew.dedetrich@aiven.io> wrote:
> Oh and I forgot to mention that I was using JDK 8 for the benchmarks since
> this
> is where the Scala2 inliner makes different impacts, latest version of
> JDK's should
> do a better job of inlining problematic bytecode generated by Scala but a
> significant
> part of our user base is still running JDK8.
>
> On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
> matthew.dedetrich@aiven.io> wrote:
>
>> Since the inliner has been merged into both Pekko and Pekko Http I
>> decided to post some results for pekko-http's jmh bench when all of the
>> inlining settings are applied (left is without inliner, right is with
>> inliner)
>>
>> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
>> Against Pekko 1.0.0-M1 (current main):
>> https://www.diffchecker.com/XhyEC556/
>>
>> As you can see from the results there are some noticeable improvements
>> (i.e. 5-10% in some cases) however I wouldn't take these results as
>> complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
>> power plus used TGPro to put fans on max blast to reduce any variability,
>> unfortunately I am currently overseas so I don't have a dedicated machine
>> to test on).
>>
>> This testing was a result of valid concerns raised by Johannes[1].
>> Another valid point being raised is about being diligent when it comes to
>> performance/benchmarks which is what dedicated hardware for benchmarking
>> was meant to solve. PJ Fanning has set up an open collective for Pekko[2]
>> and there is already a github discussion[3] about it.
>>
>> So maybe now is a good time to start talking about funding for the Pekko
>> project in general as well as being a call to action to any heavy users of
>> Pekko that feel like donating? The initial plan was to use the funds to
>> procure dedicated HW from hetzner (https://www.hetzner.com/) with
>> dedicated bare metal machines costing 50-100 euros a month, with one
>> machine being used for nightly builds and another machine dedicated to
>> benchmarking.
>>
>> We can also discuss what the additional funds can be used for (i.e.
>> sponsoring core maintainers, talks/conferences once Pekko gets big enough
>> etc etc?) although that might be best for another thread.
>>
>> Thoughts?
>>
>> [1]:
>> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
>> [2]: https://opencollective.com/friends-of-apache-pekko
>> [3]: https://github.com/apache/incubator-pekko/discussions/144
>> --
>>
>> Matthew de Detrich
>>
>> *Aiven Deutschland GmbH*
>>
>> Immanuelkirchstraße 26, 10405 Berlin
>>
>> Alexanderufer 3-7, 10117 Berlin
>>
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>
>> *m:* +491603708037
>>
>> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>>
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
Oh and I forgot to mention that I was using JDK 8 for the benchmarks since
this
is where the Scala2 inliner makes different impacts, latest version of
JDK's should
do a better job of inlining problematic bytecode generated by Scala but a
significant
part of our user base is still running JDK8.
On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
matthew.dedetrich@aiven.io> wrote:
> Since the inliner has been merged into both Pekko and Pekko Http I decided
> to post some results for pekko-http's jmh bench when all of the inlining
> settings are applied (left is without inliner, right is with inliner)
>
> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> Against Pekko 1.0.0-M1 (current main):
> https://www.diffchecker.com/XhyEC556/
>
> As you can see from the results there are some noticeable improvements
> (i.e. 5-10% in some cases) however I wouldn't take these results as
> complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
> power plus used TGPro to put fans on max blast to reduce any variability,
> unfortunately I am currently overseas so I don't have a dedicated machine
> to test on).
>
> This testing was a result of valid concerns raised by Johannes[1].
> Another valid point being raised is about being diligent when it comes to
> performance/benchmarks which is what dedicated hardware for benchmarking
> was meant to solve. PJ Fanning has set up an open collective for Pekko[2]
> and there is already a github discussion[3] about it.
>
> So maybe now is a good time to start talking about funding for the Pekko
> project in general as well as being a call to action to any heavy users of
> Pekko that feel like donating? The initial plan was to use the funds to
> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> dedicated bare metal machines costing 50-100 euros a month, with one
> machine being used for nightly builds and another machine dedicated to
> benchmarking.
>
> We can also discuss what the additional funds can be used for (i.e.
> sponsoring core maintainers, talks/conferences once Pekko gets big enough
> etc etc?) although that might be best for another thread.
>
> Thoughts?
>
> [1]:
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> [2]: https://opencollective.com/friends-of-apache-pekko
> [3]: https://github.com/apache/incubator-pekko/discussions/144
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io
Re: Inliner results for Pekko Http + Dedicated HW for benchmarking
Posted by Matthew de Detrich <ma...@aiven.io.INVALID>.
Oh and I forgot to mention that I was using JDK 8 for the benchmarks since
this
is where the Scala2 inliner makes different impacts, latest version of
JDK's should
do a better job of inlining problematic bytecode generated by Scala but a
significant
part of our user base is still running JDK8.
On Wed, Jan 17, 2024 at 12:07 PM Matthew de Detrich <
matthew.dedetrich@aiven.io> wrote:
> Since the inliner has been merged into both Pekko and Pekko Http I decided
> to post some results for pekko-http's jmh bench when all of the inlining
> settings are applied (left is without inliner, right is with inliner)
>
> Against Pekko 1.0.2: https://www.diffchecker.com/iH0WbWNZ/
> Against Pekko 1.0.0-M1 (current main):
> https://www.diffchecker.com/XhyEC556/
>
> As you can see from the results there are some noticeable improvements
> (i.e. 5-10% in some cases) however I wouldn't take these results as
> complete gospel as I had to do the benchmarks on my M1 laptop (I had it in
> power plus used TGPro to put fans on max blast to reduce any variability,
> unfortunately I am currently overseas so I don't have a dedicated machine
> to test on).
>
> This testing was a result of valid concerns raised by Johannes[1].
> Another valid point being raised is about being diligent when it comes to
> performance/benchmarks which is what dedicated hardware for benchmarking
> was meant to solve. PJ Fanning has set up an open collective for Pekko[2]
> and there is already a github discussion[3] about it.
>
> So maybe now is a good time to start talking about funding for the Pekko
> project in general as well as being a call to action to any heavy users of
> Pekko that feel like donating? The initial plan was to use the funds to
> procure dedicated HW from hetzner (https://www.hetzner.com/) with
> dedicated bare metal machines costing 50-100 euros a month, with one
> machine being used for nightly builds and another machine dedicated to
> benchmarking.
>
> We can also discuss what the additional funds can be used for (i.e.
> sponsoring core maintainers, talks/conferences once Pekko gets big enough
> etc etc?) although that might be best for another thread.
>
> Thoughts?
>
> [1]:
> https://github.com/apache/incubator-pekko-http/pull/418#issuecomment-1893375856
> [2]: https://opencollective.com/friends-of-apache-pekko
> [3]: https://github.com/apache/incubator-pekko/discussions/144
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetrich@aiven.io
>
--
Matthew de Detrich
*Aiven Deutschland GmbH*
Immanuelkirchstraße 26, 10405 Berlin
Alexanderufer 3-7, 10117 Berlin
Amtsgericht Charlottenburg, HRB 209739 B
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
*m:* +491603708037
*w:* aiven.io *e:* matthew.dedetrich@aiven.io