You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by rc...@apache.org on 2020/02/28 03:12:30 UTC
[james-project] branch master updated (d30cc4a -> e3d4734)
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from d30cc4a Provide java method for getting the events in History
new 4cab0d9 JAMES-3077 Clean up containers before running benchmarks
new c5dad9b JAMES-3077 Skip javadoc generation for benchmarks
new a061fa5 JAMES-3077 Ignore MANIFEST.MF in RAT
new 33c0507 JAMES-3059 Instrument regular statements for testing purposes
new 39bddb6 JAMES-3054 Writing document for Apache James' Mail Queue
new e3d4734 JAMES-3054 Document RabbitMQ mailQueue limitations
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
JenkinsfileStressTests.groovy | 3 +-
.../apache/james/backends/cassandra/Scenario.java | 12 ++++--
.../backends/cassandra/TestingSessionTest.java | 17 ++++++++
mailet/pom.xml | 1 +
.../server/manage-guice-distributed-james.md | 48 +++++++++++++++++++++-
5 files changed, 76 insertions(+), 5 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 05/06: JAMES-3054 Writing document for Apache
James' Mail Queue
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 39bddb6622c5c20cae0baac3fd6a9e8d363e4537
Author: Nguyen Son Tung Tran <ns...@linagora.com>
AuthorDate: Fri Feb 21 10:36:48 2020 +0700
JAMES-3054 Writing document for Apache James' Mail Queue
---
.../server/manage-guice-distributed-james.md | 41 +++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/src/site/markdown/server/manage-guice-distributed-james.md b/src/site/markdown/server/manage-guice-distributed-james.md
index 03c1a80..b00c3cf 100644
--- a/src/site/markdown/server/manage-guice-distributed-james.md
+++ b/src/site/markdown/server/manage-guice-distributed-james.md
@@ -23,7 +23,8 @@ advanced users.
- [Solving cassandra inconsistencies](#Solving_cassandra_inconsistencies)
- [Setting Cassandra user permissions](#Setting_Cassandra_user_permissions)
- [Cassandra table level configuration](#Cassandra_table_level_configuration)
-
+ - [Mail Queue](#Mail_Queue)
+
## Overall architecture
Guice distributed James server intends to provide a horizontally scalable email server.
@@ -447,3 +448,41 @@ A review of your usage can be conducted using
Table level options can be changed using **ALTER TABLE** for example with the
[cqlsh](https://cassandra.apache.org/doc/latest/tools/cqlsh.html) utility. A full compaction might be
needed in order for the changes to be taken into account.
+
+## Mail Queue
+
+An email queue is a mandatory component of SMTP servers. It is a system that creates a queue of emails that are waiting to be processed for delivery. Email queuing is a form of Message Queuing – an asynchronous service-to-service communication. A message queue is meant to decouple a producing process from a consuming one. An email queue decouples email reception from email processing. It allows them to communicate without being connected. As such, the queued emails wait for processing un [...]
+
+### Why Mail Queue is necessary
+
+You might often need to check mail queue to make sure all emails are delivered properly. At first, you need to know why email queues get clogged. Here are the two core reasons for that:
+
+- Exceeded volume of emails
+
+Some mailbox providers enforce email rate limits on IP addresses. The limits are based on the sender reputation. If you exceeded this rate and queued too many emails, the delivery speed will decrease.
+
+- Spam-related issues
+
+Another common reason is that your email has been busted by spam filters. The filters will let the emails gradually pass to analyze how the rest of the recipients react to the message. If there is slow progress, it’s okay. Your email campaign is being observed and assessed. If it’s stuck, there could be different reasons including the blockage of your IP address.
+
+### Why combining Cassandra, RabbitMQ and Object storage for MailQueue
+
+ - RabbitMQ ensures the messaging function, and avoids polling.
+ - Cassandra enables administrative operations such as browsing, deleting using a time series which might require fine performance tuning (see [Operating Casandra documentation](http://cassandra.apache.org/doc/latest/operating/index.html)).
+ - Object Storage stores potentially large binary payload.
+
+### Fine tune configuration for RabbitMQ
+
+In order to adapt mail queue settings to the actual traffic load, an administrator needs to perform fine configuration tunning as explained in [rabbitmq.properties](https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-rabbitmq.xml).
+
+### Managing email queues
+
+Managing an email queue is an easy task if you follow this procedure:
+
+- First, [List mail queues](manage-webadmin.html#Listing_mail_queues) and [get a mail queue details](manage-webadmin.html#Getting_a_mail_queue_details).
+- And then [List the mails of a mail queue](manage-webadmin.html#Listing_the_mails_of_a_mail_queue).
+- If all mails in the mail queue are needed to be delivered you will [flush mails from a mail queue](manage-webadmin.html#Flushing_mails_from_a_mail_queue).
+
+In case, you need to clear an email queue because there are only spam or trash emails in the email queue you have this procedure to follow:
+
+- All mails from the given mail queue will be deleted with [Clearing a mail queue](manage-webadmin.html#Clearing_a_mail_queue).
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 01/06: JAMES-3077 Clean up containers before
running benchmarks
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4cab0d95a4f2cbdd4f2b7edd652704d96fa6e6e9
Author: Gautier DI FOLCO <gd...@linagora.com>
AuthorDate: Wed Feb 26 11:48:17 2020 +0100
JAMES-3077 Clean up containers before running benchmarks
---
JenkinsfileStressTests.groovy | 1 +
1 file changed, 1 insertion(+)
diff --git a/JenkinsfileStressTests.groovy b/JenkinsfileStressTests.groovy
index 38d7d10..ae751c0 100644
--- a/JenkinsfileStressTests.groovy
+++ b/JenkinsfileStressTests.groovy
@@ -50,6 +50,7 @@ pipeline {
stage('Start James') {
steps {
script {
+ sh 'docker rm -f cassandra rabbitmq elasticsearch tika swift james_run || true'
if (fileExists('/srv/bench-running-docker')) {
echo 'Last build failed, cleaning provisionning'
sh 'sudo btrfs subvolume delete /srv/bench-running-docker'
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 06/06: JAMES-3054 Document RabbitMQ mailQueue
limitations
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e3d4734dcaec1b426f2d689fe02a8e183883702b
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Feb 27 17:42:14 2020 +0700
JAMES-3054 Document RabbitMQ mailQueue limitations
---
src/site/markdown/server/manage-guice-distributed-james.md | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/src/site/markdown/server/manage-guice-distributed-james.md b/src/site/markdown/server/manage-guice-distributed-james.md
index b00c3cf..0bf4c3b 100644
--- a/src/site/markdown/server/manage-guice-distributed-james.md
+++ b/src/site/markdown/server/manage-guice-distributed-james.md
@@ -471,10 +471,17 @@ Another common reason is that your email has been busted by spam filters. The fi
- Cassandra enables administrative operations such as browsing, deleting using a time series which might require fine performance tuning (see [Operating Casandra documentation](http://cassandra.apache.org/doc/latest/operating/index.html)).
- Object Storage stores potentially large binary payload.
+However the current design do not implement delays. Delays allow to define the time a mail have to be living in the
+mailqueue before being dequeued and is used for example for exponential wait delays upon remote delivery retries, or
+SMTP traffic rate limiting.
+
### Fine tune configuration for RabbitMQ
In order to adapt mail queue settings to the actual traffic load, an administrator needs to perform fine configuration tunning as explained in [rabbitmq.properties](https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-rabbitmq.xml).
+Be aware that `MailQueue::getSize` is currently performing a browse and thus is expensive. Size recurring metric
+reporting thus introduces performance issues. As such, we advise setting `mailqueue.size.metricsEnabled=false`.
+
### Managing email queues
Managing an email queue is an easy task if you follow this procedure:
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 03/06: JAMES-3077 Ignore MANIFEST.MF in RAT
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit a061fa56b35d8dac3ba7cef9f7192eeedf1d9c5e
Author: Gautier DI FOLCO <gd...@linagora.com>
AuthorDate: Wed Feb 26 14:24:06 2020 +0100
JAMES-3077 Ignore MANIFEST.MF in RAT
---
mailet/pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/mailet/pom.xml b/mailet/pom.xml
index 800019f..931041f 100644
--- a/mailet/pom.xml
+++ b/mailet/pom.xml
@@ -82,6 +82,7 @@
<exclude>NOTICE.*</exclude>
<exclude>LICENSE.*</exclude>
<!-- Generated by Maven -->
+ <exclude>**/MANIFEST.MF</exclude>
<exclude>release.properties</exclude>
<exclude>dist/**/*</exclude>
<!-- Eclipse -->
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 02/06: JAMES-3077 Skip javadoc generation for
benchmarks
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit c5dad9b1068b62187d6236f0c3ca1ae720faeb9e
Author: Gautier DI FOLCO <gd...@linagora.com>
AuthorDate: Wed Feb 26 13:39:40 2020 +0100
JAMES-3077 Skip javadoc generation for benchmarks
---
JenkinsfileStressTests.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/JenkinsfileStressTests.groovy b/JenkinsfileStressTests.groovy
index ae751c0..fead415 100644
--- a/JenkinsfileStressTests.groovy
+++ b/JenkinsfileStressTests.groovy
@@ -31,7 +31,7 @@ pipeline {
stages {
stage('Compile') {
steps {
- sh "mvn clean install -T1C -DskipTests"
+ sh "mvn clean install -T1C -DskipTests -Dmaven.javadoc.skip=true"
}
}
stage('Build image') {
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 04/06: JAMES-3059 Instrument regular statements for
testing purposes
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 33c0507a05fcc77af16cae77ec01d60f591b9ebe
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Feb 27 15:00:40 2020 +0700
JAMES-3059 Instrument regular statements for testing purposes
---
.../org/apache/james/backends/cassandra/Scenario.java | 12 +++++++++---
.../james/backends/cassandra/TestingSessionTest.java | 17 +++++++++++++++++
2 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
index 4bc8746..efbf6a8 100644
--- a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
+++ b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
@@ -26,6 +26,7 @@ import java.util.stream.Stream;
import org.testcontainers.shaded.com.google.common.collect.ImmutableList;
import com.datastax.driver.core.BoundStatement;
+import com.datastax.driver.core.RegularStatement;
import com.datastax.driver.core.ResultSetFuture;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.Statement;
@@ -54,10 +55,10 @@ public class Scenario {
@FunctionalInterface
interface StatementPredicate {
- class BoundStatementStartingWith implements StatementPredicate {
+ class StatementStartingWith implements StatementPredicate {
private final String queryStringPrefix;
- BoundStatementStartingWith(String queryStringPrefix) {
+ StatementStartingWith(String queryStringPrefix) {
this.queryStringPrefix = queryStringPrefix;
}
@@ -69,6 +70,11 @@ public class Scenario {
.getQueryString()
.startsWith(queryStringPrefix);
}
+ if (statement instanceof RegularStatement) {
+ RegularStatement regularStatement = (RegularStatement) statement;
+ return regularStatement.getQueryString()
+ .startsWith(queryStringPrefix);
+ }
return false;
}
}
@@ -122,7 +128,7 @@ public class Scenario {
}
default ExecutionHook whenQueryStartsWith(String queryStringPrefix) {
- return statementPredicate(new StatementPredicate.BoundStatementStartingWith(queryStringPrefix));
+ return statementPredicate(new StatementPredicate.StatementStartingWith(queryStringPrefix));
}
}
diff --git a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
index 941c16b..bb08907 100644
--- a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
+++ b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
@@ -19,15 +19,19 @@
package org.apache.james.backends.cassandra;
+import static com.datastax.driver.core.querybuilder.QueryBuilder.select;
import static org.apache.james.backends.cassandra.Scenario.Builder.awaitOn;
import static org.apache.james.backends.cassandra.Scenario.Builder.executeNormally;
import static org.apache.james.backends.cassandra.Scenario.Builder.fail;
import static org.apache.james.backends.cassandra.Scenario.combine;
+import static org.apache.james.backends.cassandra.versions.table.CassandraSchemaVersionTable.TABLE_NAME;
+import static org.apache.james.backends.cassandra.versions.table.CassandraSchemaVersionTable.VALUE;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import org.apache.james.backends.cassandra.Scenario.Barrier;
+import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionDAO;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule;
import org.apache.james.backends.cassandra.versions.SchemaVersion;
@@ -90,6 +94,19 @@ class TestingSessionTest {
}
@Test
+ void regularStatementsShouldBeInstrumented(CassandraCluster cassandra) {
+ cassandra.getConf()
+ .registerScenario(fail()
+ .times(1)
+ .whenQueryStartsWith("SELECT value FROM schemaVersion;"));
+
+ assertThatThrownBy(() -> new CassandraAsyncExecutor(cassandra.getConf())
+ .execute(select(VALUE).from(TABLE_NAME))
+ .block())
+ .isInstanceOf(RuntimeException.class);
+ }
+
+ @Test
void forAllQueriesShouldMatchAllStatements(CassandraCluster cassandra) {
cassandra.getConf()
.registerScenario(fail()
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org