You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Benoit TELLIER <bt...@apache.org> on 2022/05/23 05:53:58 UTC
Upcoming "polish" contributions (June-July 2022)
Hello dear community,
With my team we plan to invest ~ one month polishing James.
We will invest some of our time to hopefully address some technical-dept
related topic for the distributed James server.
So far this includes:
*# Reactor elastic -> boundedElastic switch*
A scheduler is the abstraction Reactor, our reactive library runs code
on. Ideally we would only run non-blocking code on a parallel scheduler.
In practice we also run blocking code on an elastic scheduler. Elastic
schedulers are unbounded so we can end up with a very large number of
threads thus defeating the benefits of reactive code.
For this reason, project reactor deprecates elastic in favor of bounded
elastic.
This switch was attempted before but resulted in dead locks.
Such deadlocks can be avoided by:
- Avoiding wrapping .blocks calls in an elastic thread. This means
essentially improving the quality of our reactive pipelines and not
blocking them in the middle.
- Avoid calls to subscibeOn before blocking. Instead use the blocked
threads (that is anyway blocked) for the ongoing computation.
- In last result use a SEDA like approach:
- Introduce a new instance of boundedElastic dedicated to wrap our
calls to .block where we can't avoid it.
- Use core boundedElastic only when no reactor .block call is
involved to prevent starvation
Alternatives would be to increase to an absurd amount the size of
boundedElastic essentially mimicing the behaviour of elastic.
We have 161 calls to .elastic().
Acceptance criteria: no performance degradation. Recent performance
improvments were furthermore delivered in recent reactor releases.
This would eventually be needed to upgrade to reactor 3.5.0.
*
**# Cassandra driver 4*
We rely on the Cassandra driver 3, which is technically still maintained
but only for security purposes.
The biggest downside of using this driver is that it is not truly reactive:
- Bridging through completable futures make us loose back-preassure
- Even worse: paging large datasets is blocking which forces us to
aggressively subscribe on an other thread to prevent blocking calls on
the Cassandra driver event loop.
This means dozens of context switches per IMAP queries (eg). Flame
graphs shows we spend ~10% switching threads.
Those drawbacks would eventually disappear with Cassandra driver 4,
offering native reactive constructs.
We expect side benefits like usage of protocol version 5, as well as an
overall improvement in the driver quality, which would translate to a
nice performance boost.
Work was conducted on this: https://github.com/apache/james-project/pull/599
Impact:
- Switch Cassandra configuration to the Cassandra driver 4 semantic.
This means users will need to adapt their configuration.
- Boiler plate changes everywhere. Though this should be reasonably
easy. This should be transparent to users, unless they uses Cassandra
driver in some of their extensions.
*# Upgrade to ElasticSearch 8.2*
Because 7.10 is end of life... https://www.elastic.co/support/eol
We will perform the classic switch by copying the code and moving things
one by one.
Impact: users will need to upgrade their ElasticSearch cluster too!
*
**# General dependency upgrades*
Sanitize our dependencies, plugins, docker images and upgrade everything
that can be.
Doing so tend to reduce security risk, allows to diagnose unmaintained
dependencies, can fix some pain points and add performance.
A must do!
Last time was 6 months ago so this should be globally quick!
*# ElasticSearch indexing**
*
We identified indexation to happen at a reasonably slow pace, at ~50
messages per seconds which is slow.
Most likely this is due to the use of NESTED semantic in header
indexing. This enables us a static indexing while preserving search
semantic, but at the cost of inserting one extra document per mail
header, resulting in very large document count.
We could experiment alternative header indexation strategies that could
allow gains.
For instance I putted together something relying on FLATTENED semantic:
https://github.com/apache/james-project/pull/988
We would take a bit of time to back those proposal with metrics.
*
**# Others...*
Those are the topics we identified but we might have had missed some!
Do you think some other topics would fit such a polish?
By the way, help would greatly be welcomed on such topics!
Regards,
Benoit
Re: Upcoming "polish" contributions (June-July 2022)
Posted by Rene Cordier <rc...@apache.org>.
Hello,
Just to quickly come back to the ElasticSearch upgrade to 8.2 as I
started taking a look.
It seems we gonna need to migrate the api client used in our code as the
version 8 comes with an entirely new client library:
https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/migrate-hlrc.html
Let's hope it performs better and that it reduces the number of
transitive dependencies it added on the classpath.
It might take longer to migrate though, but we hope to get some good
benefits out of it.
Cheers,
Rene.
On 23/05/2022 12:53, Benoit TELLIER wrote:
> Hello dear community,
>
> With my team we plan to invest ~ one month polishing James.
>
> We will invest some of our time to hopefully address some technical-dept
> related topic for the distributed James server.
>
> So far this includes:
>
> *# Reactor elastic -> boundedElastic switch*
>
> A scheduler is the abstraction Reactor, our reactive library runs code
> on. Ideally we would only run non-blocking code on a parallel scheduler.
> In practice we also run blocking code on an elastic scheduler. Elastic
> schedulers are unbounded so we can end up with a very large number of
> threads thus defeating the benefits of reactive code.
>
> For this reason, project reactor deprecates elastic in favor of bounded
> elastic.
>
> This switch was attempted before but resulted in dead locks.
>
> Such deadlocks can be avoided by:
> - Avoiding wrapping .blocks calls in an elastic thread. This means
> essentially improving the quality of our reactive pipelines and not
> blocking them in the middle.
> - Avoid calls to subscibeOn before blocking. Instead use the blocked
> threads (that is anyway blocked) for the ongoing computation.
> - In last result use a SEDA like approach:
> - Introduce a new instance of boundedElastic dedicated to wrap our
> calls to .block where we can't avoid it.
> - Use core boundedElastic only when no reactor .block call is
> involved to prevent starvation
>
> Alternatives would be to increase to an absurd amount the size of
> boundedElastic essentially mimicing the behaviour of elastic.
>
> We have 161 calls to .elastic().
>
> Acceptance criteria: no performance degradation. Recent performance
> improvments were furthermore delivered in recent reactor releases.
>
> This would eventually be needed to upgrade to reactor 3.5.0.
> *
> **# Cassandra driver 4*
>
> We rely on the Cassandra driver 3, which is technically still maintained
> but only for security purposes.
>
> The biggest downside of using this driver is that it is not truly reactive:
> - Bridging through completable futures make us loose back-preassure
> - Even worse: paging large datasets is blocking which forces us to
> aggressively subscribe on an other thread to prevent blocking calls on
> the Cassandra driver event loop.
>
> This means dozens of context switches per IMAP queries (eg). Flame
> graphs shows we spend ~10% switching threads.
>
> Those drawbacks would eventually disappear with Cassandra driver 4,
> offering native reactive constructs.
>
> We expect side benefits like usage of protocol version 5, as well as an
> overall improvement in the driver quality, which would translate to a
> nice performance boost.
>
> Work was conducted on this:
> https://github.com/apache/james-project/pull/599
>
> Impact:
> - Switch Cassandra configuration to the Cassandra driver 4 semantic.
> This means users will need to adapt their configuration.
> - Boiler plate changes everywhere. Though this should be reasonably
> easy. This should be transparent to users, unless they uses Cassandra
> driver in some of their extensions.
>
> *# Upgrade to ElasticSearch 8.2*
>
> Because 7.10 is end of life... https://www.elastic.co/support/eol
>
> We will perform the classic switch by copying the code and moving things
> one by one.
>
> Impact: users will need to upgrade their ElasticSearch cluster too!
> *
> **# General dependency upgrades*
>
> Sanitize our dependencies, plugins, docker images and upgrade everything
> that can be.
>
> Doing so tend to reduce security risk, allows to diagnose unmaintained
> dependencies, can fix some pain points and add performance.
>
> A must do!
>
> Last time was 6 months ago so this should be globally quick!
>
> *# ElasticSearch indexing**
> *
> We identified indexation to happen at a reasonably slow pace, at ~50
> messages per seconds which is slow.
>
> Most likely this is due to the use of NESTED semantic in header
> indexing. This enables us a static indexing while preserving search
> semantic, but at the cost of inserting one extra document per mail
> header, resulting in very large document count.
>
> We could experiment alternative header indexation strategies that could
> allow gains.
>
> For instance I putted together something relying on FLATTENED semantic:
> https://github.com/apache/james-project/pull/988
>
> We would take a bit of time to back those proposal with metrics.
> *
> **# Others...*
>
> Those are the topics we identified but we might have had missed some!
>
> Do you think some other topics would fit such a polish?
>
> By the way, help would greatly be welcomed on such topics!
>
> Regards,
>
> Benoit
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org