You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Quan tran hong <qu...@gmail.com> on 2024/02/28 09:08:29 UTC

Result of Proof Of Concept for Redis Event bus user notifications

Hi folks,

Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
Event bus user notifications to Redis
<https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues>
(TLDR:
we observed some annoying issues with RabbitMQ in a deployment and we think
it could be better to move at least the user notifications part of Event
Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
Redis event bus POC <https://github.com/apache/james-project/pull/2028>.

The POC is considered done to me, and I want to share the POC result to
James devs:
- It is *possible *(the POC worked!) to replace the Event bus user
notifications (key registration) using Redis Pub/Sub. And likely the whole
EventBus (the group registration part as well) can rely on Redis (I did not
do that part).
- I did several performance tests for the POC, and the results were *good*.
Regarding the metrics of Event Bus user notifications, Redis seems to
outperform and show more stability in response time than RabbitMQ. (For
more details, I shared on the PR)

Despite the POC has been worked and shown some prospects, I think we should
monitor it more carefully before applying it to James. Therefore, we
(Linagora) would adopt the Redis Event bus user notifications first in our
TMail project (based on James). We would keep an eye on its stability
before contributing the fine-tuned work toward James.

By the way, it seems someone from the community is interested in building
the whole EventBus using Redis cf
https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a start
for that Redis EventBus.

What do you think about using Redis for EventBus? Would that sound
interesting to you?

Regards,

Quan

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Maksim Meliashchuk <ma...@gmail.com>.
Hello! I'm interested in this topic. I prefer to consider the subtleties
that experts talk about. If there is an opinion that it is not desirable to
expand the use of Redis, then
https://issues.apache.org/jira/browse/JAMES-3956 can be closed. But if
anyone can provide assistance in defining the task and decomposing it, then
I may continue to do something extra for notifications/registration.

ср, 28 февр. 2024 г. в 20:59, Benoit TELLIER <bt...@apache.org>:

> Hello Quan,
>
> Thanks a lot for this work. It is very encouraging.
>
> Having the whole eventbus based on Redis is IMO however not a good idea.
>
> For notifications / registration being best effort will be OK however
> RabbitMQ quorum queues address reliability issues. Going fully away of
> RabbitMQ for work queue patterns in favor of Redis is IMO not desirable.
>
> Best regards,
>
> Benoit TELLIER
>
> On 28/02/2024 10:08, Quan tran hong wrote:
> > Hi folks,
> >
> > Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> > Event bus user notifications to Redis
> > <
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
> >
> > (TLDR:
> > we observed some annoying issues with RabbitMQ in a deployment and we
> think
> > it could be better to move at least the user notifications part of Event
> > Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> > Redis event bus POC <https://github.com/apache/james-project/pull/2028>.
> >
> > The POC is considered done to me, and I want to share the POC result to
> > James devs:
> > - It is *possible *(the POC worked!) to replace the Event bus user
> > notifications (key registration) using Redis Pub/Sub. And likely the
> whole
> > EventBus (the group registration part as well) can rely on Redis (I did
> not
> > do that part).
> > - I did several performance tests for the POC, and the results were
> *good*.
> > Regarding the metrics of Event Bus user notifications, Redis seems to
> > outperform and show more stability in response time than RabbitMQ. (For
> > more details, I shared on the PR)
> >
> > Despite the POC has been worked and shown some prospects, I think we
> should
> > monitor it more carefully before applying it to James. Therefore, we
> > (Linagora) would adopt the Redis Event bus user notifications first in
> our
> > TMail project (based on James). We would keep an eye on its stability
> > before contributing the fine-tuned work toward James.
> >
> > By the way, it seems someone from the community is interested in building
> > the whole EventBus using Redis cf
> > https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
> start
> > for that Redis EventBus.
> >
> > What do you think about using Redis for EventBus? Would that sound
> > interesting to you?
> >
> > Regards,
> >
> > Quan
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Benoit TELLIER <bt...@apache.org>.
Hello Quan,

Thanks a lot for this work. It is very encouraging.

Having the whole eventbus based on Redis is IMO however not a good idea.

For notifications / registration being best effort will be OK however 
RabbitMQ quorum queues address reliability issues. Going fully away of 
RabbitMQ for work queue patterns in favor of Redis is IMO not desirable.

Best regards,

Benoit TELLIER

On 28/02/2024 10:08, Quan tran hong wrote:
> Hi folks,
>
> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> Event bus user notifications to Redis
> <https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues>
> (TLDR:
> we observed some annoying issues with RabbitMQ in a deployment and we think
> it could be better to move at least the user notifications part of Event
> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> Redis event bus POC <https://github.com/apache/james-project/pull/2028>.
>
> The POC is considered done to me, and I want to share the POC result to
> James devs:
> - It is *possible *(the POC worked!) to replace the Event bus user
> notifications (key registration) using Redis Pub/Sub. And likely the whole
> EventBus (the group registration part as well) can rely on Redis (I did not
> do that part).
> - I did several performance tests for the POC, and the results were *good*.
> Regarding the metrics of Event Bus user notifications, Redis seems to
> outperform and show more stability in response time than RabbitMQ. (For
> more details, I shared on the PR)
>
> Despite the POC has been worked and shown some prospects, I think we should
> monitor it more carefully before applying it to James. Therefore, we
> (Linagora) would adopt the Redis Event bus user notifications first in our
> TMail project (based on James). We would keep an eye on its stability
> before contributing the fine-tuned work toward James.
>
> By the way, it seems someone from the community is interested in building
> the whole EventBus using Redis cf
> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a start
> for that Redis EventBus.
>
> What do you think about using Redis for EventBus? Would that sound
> interesting to you?
>
> Regards,
>
> Quan
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Benoit TELLIER <bt...@apache.org>.
Hello Jean

On 01/03/2024 14:01, Jean Helou wrote:
> Hello Benoit
>
> This is very clear thanks. I was not implying you should use pulsar :)  I
> use a sass deployment and I am aware it is quite complex to operate:)
> I might well end up deciding that a pulsar+redis makes more sense than
> pulsar only when I start trying to deploy the MDA side of James:)
>
I like that flexibility!

Combining different technologies for queuing and pubsub likely make 
sense, but maybe we would need to have more "generic" interfaces under 
the hood to make those combinations easier to achieve.

If you are interested we could plan a "pair programming" session on the 
topic...
>
>   - Currently the James domain logic would imply one topic per mailbox
>> (more or less) and we would too quickly IMO end up with an unconfortable
>> number of topics (million+). A refactoring would be needed.
>>
> Hmm this is interesting. I will have to review if it would be problematic
> for a pulsar implem do you have a good entry point (class name(s)) I should
> look at to investigate ?
Well the problem is more linked to the use of the eventbus than the 
event bus itself.

https://github.com/apache/james-project/blob/master/mailbox/api/src/main/java/org/apache/james/mailbox/events/MailboxIdRegistrationKey.java 

https://github.com/apache/james-project/blob/d3ec938e0aab741b79c9199e6118ee5464ead646/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java#L316

>
> Thanks again for the explanation
>
> Cheers
> Jean
>
>   Note that all of that makes it a Linagora topic, at least now. If
>> relevant, we would happily contribute it back ;-)
>>
>> Is this more clear?
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> On 01/03/2024 13:17, Jean Helou wrote:
>>> Hello 👋
>>>
>>> I'm not (yet) familiar with the event bus :)
>>>
>>> Out of curiosity could you describe what made you chose redis and what
>>> other systems you considered ?
>>> What characteristics you thought would match well with the problem.
>>>
>>> I'm interested in part because there is a Jira to implement the event bus
>>> over pulsar so I'd like to take the opportunity to better understand the
>>> design ideas behind the event bus as that would help with the pulsar
>>> implementation.
>>>
>>> Jean
>>>
>>>
>>> Le mer. 28 févr. 2024 à 10:09, Quan tran hong <
>> quan.tranhong1999@gmail.com>
>>> a écrit :
>>>
>>>> Hi folks,
>>>>
>>>> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
>>>> Event bus user notifications to Redis
>>>> <
>>>>
>> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
>>>> (TLDR:
>>>> we observed some annoying issues with RabbitMQ in a deployment and we
>> think
>>>> it could be better to move at least the user notifications part of Event
>>>> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
>>>> Redis event bus POC <https://github.com/apache/james-project/pull/2028
>>> .
>>>> The POC is considered done to me, and I want to share the POC result to
>>>> James devs:
>>>> - It is *possible *(the POC worked!) to replace the Event bus user
>>>> notifications (key registration) using Redis Pub/Sub. And likely the
>> whole
>>>> EventBus (the group registration part as well) can rely on Redis (I did
>> not
>>>> do that part).
>>>> - I did several performance tests for the POC, and the results were
>> *good*.
>>>> Regarding the metrics of Event Bus user notifications, Redis seems to
>>>> outperform and show more stability in response time than RabbitMQ. (For
>>>> more details, I shared on the PR)
>>>>
>>>> Despite the POC has been worked and shown some prospects, I think we
>> should
>>>> monitor it more carefully before applying it to James. Therefore, we
>>>> (Linagora) would adopt the Redis Event bus user notifications first in
>> our
>>>> TMail project (based on James). We would keep an eye on its stability
>>>> before contributing the fine-tuned work toward James.
>>>>
>>>> By the way, it seems someone from the community is interested in
>> building
>>>> the whole EventBus using Redis cf
>>>> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
>>>> start
>>>> for that Redis EventBus.
>>>>
>>>> What do you think about using Redis for EventBus? Would that sound
>>>> interesting to you?
>>>>
>>>> Regards,
>>>>
>>>> Quan
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Jean Helou <jh...@apache.org>.
Hello Benoit

This is very clear thanks. I was not implying you should use pulsar :)  I
use a sass deployment and I am aware it is quite complex to operate:)
I might well end up deciding that a pulsar+redis makes more sense than
pulsar only when I start trying to deploy the MDA side of James:)

 - Currently the James domain logic would imply one topic per mailbox
> (more or less) and we would too quickly IMO end up with an unconfortable
> number of topics (million+). A refactoring would be needed.
>

Hmm this is interesting. I will have to review if it would be problematic
for a pulsar implem do you have a good entry point (class name(s)) I should
look at to investigate ?

Thanks again for the explanation

Cheers
Jean

 Note that all of that makes it a Linagora topic, at least now. If
> relevant, we would happily contribute it back ;-)
>
> Is this more clear?
>
> Best regards,
>
> Benoit TELLIER
>
> On 01/03/2024 13:17, Jean Helou wrote:
> > Hello 👋
> >
> > I'm not (yet) familiar with the event bus :)
> >
> > Out of curiosity could you describe what made you chose redis and what
> > other systems you considered ?
> > What characteristics you thought would match well with the problem.
> >
> > I'm interested in part because there is a Jira to implement the event bus
> > over pulsar so I'd like to take the opportunity to better understand the
> > design ideas behind the event bus as that would help with the pulsar
> > implementation.
> >
> > Jean
> >
> >
> > Le mer. 28 févr. 2024 à 10:09, Quan tran hong <
> quan.tranhong1999@gmail.com>
> > a écrit :
> >
> >> Hi folks,
> >>
> >> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> >> Event bus user notifications to Redis
> >> <
> >>
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
> >> (TLDR:
> >> we observed some annoying issues with RabbitMQ in a deployment and we
> think
> >> it could be better to move at least the user notifications part of Event
> >> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> >> Redis event bus POC <https://github.com/apache/james-project/pull/2028
> >.
> >>
> >> The POC is considered done to me, and I want to share the POC result to
> >> James devs:
> >> - It is *possible *(the POC worked!) to replace the Event bus user
> >> notifications (key registration) using Redis Pub/Sub. And likely the
> whole
> >> EventBus (the group registration part as well) can rely on Redis (I did
> not
> >> do that part).
> >> - I did several performance tests for the POC, and the results were
> *good*.
> >> Regarding the metrics of Event Bus user notifications, Redis seems to
> >> outperform and show more stability in response time than RabbitMQ. (For
> >> more details, I shared on the PR)
> >>
> >> Despite the POC has been worked and shown some prospects, I think we
> should
> >> monitor it more carefully before applying it to James. Therefore, we
> >> (Linagora) would adopt the Redis Event bus user notifications first in
> our
> >> TMail project (based on James). We would keep an eye on its stability
> >> before contributing the fine-tuned work toward James.
> >>
> >> By the way, it seems someone from the community is interested in
> building
> >> the whole EventBus using Redis cf
> >> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
> >> start
> >> for that Redis EventBus.
> >>
> >> What do you think about using Redis for EventBus? Would that sound
> >> interesting to you?
> >>
> >> Regards,
> >>
> >> Quan
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Benoit TELLIER <bt...@apache.org>.
Hello Jean!

First, the decision is made at Linagora. I hope that we can come up with 
enough feedback to convince people in the Apache community.

Why Redis?
  - RabbitMQ is a bad match for pub sub, I won't repeat the litany of 
issues encountered with it...
  - We are using Redis in our tech mix, at least at linagora:
    - Backend for our OIDC backchannel logout
    - Distributed rate limiter
    - Backend for RSpamd rules
  - Cost of operation matters to us. We are on a mostly on-premise model 
and need to be profitable with ~5-10.000 users which means the money to 
pay for the infra bill is limited. We see Redis+Rabbit mix as cheaper to 
operate than Pulsar, at least for medium sized deployments.
  - We do not master pulsar, so there would be a learning curve.
  - Having only the "notification" part of the eventbus on Redis means 
we can tolerate unavalability /data loss: it is not that critical.
  - Currently the James domain logic would imply one topic per mailbox 
(more or less) and we would too quickly IMO end up with an unconfortable 
number of topics (million+). A refactoring would be needed.
    Not to mention it would ease implementing things like IMAP NOTIFY.
  - Timing: I need to provide a HA solution "ASAP" for a production 
platform. The "Redis" plan can be implemented in a shorter time frame 
than the "pulsar"one.

Note that "queuing" topics are not to be handled with Redis.

Note that all of that makes it a Linagora topic, at least now. If 
relevant, we would happily contribute it back ;-)

Is this more clear?

Best regards,

Benoit TELLIER

On 01/03/2024 13:17, Jean Helou wrote:
> Hello 👋
>
> I'm not (yet) familiar with the event bus :)
>
> Out of curiosity could you describe what made you chose redis and what
> other systems you considered ?
> What characteristics you thought would match well with the problem.
>
> I'm interested in part because there is a Jira to implement the event bus
> over pulsar so I'd like to take the opportunity to better understand the
> design ideas behind the event bus as that would help with the pulsar
> implementation.
>
> Jean
>
>
> Le mer. 28 févr. 2024 à 10:09, Quan tran hong <qu...@gmail.com>
> a écrit :
>
>> Hi folks,
>>
>> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
>> Event bus user notifications to Redis
>> <
>> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
>> (TLDR:
>> we observed some annoying issues with RabbitMQ in a deployment and we think
>> it could be better to move at least the user notifications part of Event
>> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
>> Redis event bus POC <https://github.com/apache/james-project/pull/2028>.
>>
>> The POC is considered done to me, and I want to share the POC result to
>> James devs:
>> - It is *possible *(the POC worked!) to replace the Event bus user
>> notifications (key registration) using Redis Pub/Sub. And likely the whole
>> EventBus (the group registration part as well) can rely on Redis (I did not
>> do that part).
>> - I did several performance tests for the POC, and the results were *good*.
>> Regarding the metrics of Event Bus user notifications, Redis seems to
>> outperform and show more stability in response time than RabbitMQ. (For
>> more details, I shared on the PR)
>>
>> Despite the POC has been worked and shown some prospects, I think we should
>> monitor it more carefully before applying it to James. Therefore, we
>> (Linagora) would adopt the Redis Event bus user notifications first in our
>> TMail project (based on James). We would keep an eye on its stability
>> before contributing the fine-tuned work toward James.
>>
>> By the way, it seems someone from the community is interested in building
>> the whole EventBus using Redis cf
>> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
>> start
>> for that Redis EventBus.
>>
>> What do you think about using Redis for EventBus? Would that sound
>> interesting to you?
>>
>> Regards,
>>
>> Quan
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Result of Proof Of Concept for Redis Event bus user notifications

Posted by Jean Helou <jh...@apache.org>.
Hello 👋

I'm not (yet) familiar with the event bus :)

Out of curiosity could you describe what made you chose redis and what
other systems you considered ?
What characteristics you thought would match well with the problem.

I'm interested in part because there is a Jira to implement the event bus
over pulsar so I'd like to take the opportunity to better understand the
design ideas behind the event bus as that would help with the pulsar
implementation.

Jean


Le mer. 28 févr. 2024 à 10:09, Quan tran hong <qu...@gmail.com>
a écrit :

> Hi folks,
>
> Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
> Event bus user notifications to Redis
> <
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3996?filter=allopenissues
> >
> (TLDR:
> we observed some annoying issues with RabbitMQ in a deployment and we think
> it could be better to move at least the user notifications part of Event
> Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
> Redis event bus POC <https://github.com/apache/james-project/pull/2028>.
>
> The POC is considered done to me, and I want to share the POC result to
> James devs:
> - It is *possible *(the POC worked!) to replace the Event bus user
> notifications (key registration) using Redis Pub/Sub. And likely the whole
> EventBus (the group registration part as well) can rely on Redis (I did not
> do that part).
> - I did several performance tests for the POC, and the results were *good*.
> Regarding the metrics of Event Bus user notifications, Redis seems to
> outperform and show more stability in response time than RabbitMQ. (For
> more details, I shared on the PR)
>
> Despite the POC has been worked and shown some prospects, I think we should
> monitor it more carefully before applying it to James. Therefore, we
> (Linagora) would adopt the Redis Event bus user notifications first in our
> TMail project (based on James). We would keep an eye on its stability
> before contributing the fine-tuned work toward James.
>
> By the way, it seems someone from the community is interested in building
> the whole EventBus using Redis cf
> https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a
> start
> for that Redis EventBus.
>
> What do you think about using Redis for EventBus? Would that sound
> interesting to you?
>
> Regards,
>
> Quan
>