You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by GitBox <gi...@apache.org> on 2021/06/23 08:37:49 UTC
[GitHub] [james-project] chibenwa opened a new pull request #508: JAMES-3604 Improve RabbitMQ resiliency
chibenwa opened a new pull request #508:
URL: https://github.com/apache/james-project/pull/508
- [x] Allow the use of Quorum queues for the EventBus and the MailQueues
- [x] Use publish confirms as recommended
- [ ] Allow via the configuration to pass several contact points
- [ ] Measure performance costs of publish confirms
To be done later:
- [ ] Once we deployed a RabbitMQ cluster in one of our environments at Linagora, and once we did deploy Quorum queues with a replication factor of 3, we will perform a quick return of experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a change in pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on a change in pull request #508:
URL: https://github.com/apache/james-project/pull/508#discussion_r663635870
##########
File path: server/apps/distributed-app/sample-configuration/rabbitmq.properties
##########
@@ -5,6 +5,10 @@
# Mandatory
uri=amqp://rabbitmq:5672
Review comment:
> Should we deprecate uri property?
If I can avoid a breaking change I'm happy.
I think URI is still easier to use for a single Rabbit set up.
You can further more pass extra info in it: VHost, username + password, etc...
> When we declare hosts property, what is uri sense?
1. Host is optional
2. URI can be used for vhost, username, password, etc...
> Should we have only one property for endpoint declaration? IMO
We have to balance this with other goals too: how about avoiding to much breaking changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a change in pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on a change in pull request #508:
URL: https://github.com/apache/james-project/pull/508#discussion_r663635870
##########
File path: server/apps/distributed-app/sample-configuration/rabbitmq.properties
##########
@@ -5,6 +5,10 @@
# Mandatory
uri=amqp://rabbitmq:5672
Review comment:
> Should we deprecate uri property?
If I can avoid a breaking change I'm happy.
I think URI is still easier to use for a single Rabbit set up.
You can further more pass extra info in it: VHost, username + password, SSL, etc...
> When we declare hosts property, what is uri sense?
1. Host is optional
2. URI can be used for vhost, username, password, etc...
> Should we have only one property for endpoint declaration? IMO
We have to balance this with other goals too: how about avoiding to much breaking changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] Arsnael commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
Arsnael commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-875230062
Yeah I just saw that... What can we do about it, as the corrupted m2 package is on apache jenkins?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-869344454
Undrafted as I move publish/confirms to a separated PR...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa edited a comment on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa edited a comment on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-869131638
CPU utilisation related to RabbitMQ as expected increased (1.30 % -> 2.6%) however latencies went from 67ms to 90+ ms at ~800 req/s.
RPC related to `Channel::confirmSelect` done repeatedly are rather expensive, and I notice some active creation of channels.
As such, I feel unconfident with the publish / confirm part of the work and would split it to another pull request...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] Arsnael commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
Arsnael commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-875211965
Why closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-875220740
Some build errors, got that on another PR and I was forced to reopen it to sole it. Looked like a corrupted .m2...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa closed pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa closed pull request #508:
URL: https://github.com/apache/james-project/pull/508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa closed pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa closed pull request #508:
URL: https://github.com/apache/james-project/pull/508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa edited a comment on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa edited a comment on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-869131638
CPU utilisation related to RabbitMQ as expected increased (1.30 % -> 2.6%) however latencies went from 67ms to 90+ ms at ~800 req/s.
RPC related to `Channel::confirmSelect` done repeatedly are rather expensive, and I notice some active creation of channels.
As such, I feel unconfident with the publish / confirm part of the work and would split it to another pull request...
https://www.rabbitmq.com/tutorials/tutorial-seven-java.html
```
Confirms should be enabled just once, not for every message published.
```
Not the case given my implementation.
For now I see several solutions:
- rely on a sink to pass all messages as part of a single Flux. Thus init is done once. The done side is that the result is asynchronous to the caller (error handling would not be propagated)
- enable publish confirms globally in ReactorRabbitMQChannelPool and re-implement send operation to skip confirmSelect activation...
- Wrap channel in a super-class that activated `confirmSelect` only once (AtomicBoolean?)
- ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] vttranlina commented on a change in pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
vttranlina commented on a change in pull request #508:
URL: https://github.com/apache/james-project/pull/508#discussion_r663631831
##########
File path: src/site/xdoc/server/config-rabbitmq.xml
##########
@@ -118,6 +118,11 @@
<dt><strong>quorum.queues.replication.factor</strong></dt>
<dd> Strictly positive integer. The replication factor to use when creating quorum queues.</dd>
+
+ <dt><strong>hostsr</strong></dt>
Review comment:
`hosts`
##########
File path: server/apps/distributed-app/sample-configuration/rabbitmq.properties
##########
@@ -5,6 +5,10 @@
# Mandatory
uri=amqp://rabbitmq:5672
Review comment:
Should we deprecate `uri` property?
When we declare `hosts` property, what is `uri` sense?
Should we have only one property for endpoint declaration? IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-874399857
Just squashed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] Arsnael commented on a change in pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
Arsnael commented on a change in pull request #508:
URL: https://github.com/apache/james-project/pull/508#discussion_r659556881
##########
File path: server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/AmqpForwardAttachmentTest.java
##########
@@ -71,6 +71,7 @@
@BeforeEach
void setup(@TempDir File temporaryFolder) throws Exception {
+ System.out.println(amqpExtension.getAmqpUri());
Review comment:
debug spotted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-874399857
Just squashed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #508:
URL: https://github.com/apache/james-project/pull/508#issuecomment-869131638
CPU utilisation related to RabbitMQ as expected increased (1.30 % -> 2.6%) and we lost a few ms in latencies.
Given that it is required for correctness, this sounds like a decent compromise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] vttranlina commented on a change in pull request #508: JAMES-3604 Improve RabbitMQ resiliency
Posted by GitBox <gi...@apache.org>.
vttranlina commented on a change in pull request #508:
URL: https://github.com/apache/james-project/pull/508#discussion_r663638756
##########
File path: server/apps/distributed-app/sample-configuration/rabbitmq.properties
##########
@@ -5,6 +5,10 @@
# Mandatory
uri=amqp://rabbitmq:5672
Review comment:
Thanks for answer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org