You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by bt...@apache.org on 2020/07/17 02:24:14 UTC
[james-project] branch master updated (c23267a -> 4e099d0)
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from c23267a JAMES-3308: add test in RabbitMQTerminationSubscriberTest for deserialization error handling
new 17f8fc5 JAMES-3296 Add republishing to RabbitMQMailQueue from Cassandra capability
new 7bc6a36 JAMES-3296 Add task to republish RabbitMQ MailQueue from Cassandra
new 85ba240 JAMES-3107 Log slow traces to WARN
new 7ed1980 JAMES-3305 Add FailsDeserializationTask type for testing
new aebfaa9 JAMES-3305 Task manager deserialization error handling
new d3cf49f JAMES-2904 Remove unused MessageResult::hasAttachment
new add3e9a JAMES−2290 Fix unstable test: DiscreteDistributionTest.partitionShouldSupportDuplicatedDistributionEntry
new 06431cd [REFACTORING] Avoid variable reallocation in FetchResponseBuilder
new ba3a096 [REFACTORING] FetchResponseBuilder should use Optional::isEmpty
new 3441547 [REFACTORING] MessageResultUtils never throws
new d4dd474 [REFACTORING] Remove unused methods in MessageResultUtils
new b531212 [REFACTORING] Remove unused FetchResponse.Address empty array
new 823ed4e [REFACTORING] Avoid variable reallocation in EnvelopeBuilder
new a2ba8b1 [REFACTORING] Avoid variable reallocation in PartialFetchBodyElement
new 3f1f171 [REFACTORING] IMAP FETCH javaDoc fixes
new 0779dea [REFACTORING] MimeBodyElement should use StandardCharset
new 83b4d2b [REFACTORING] Rearrange IMAP FETCH fields
new a78237b [REFACTORING] Remove uneeded else blocks in IMAP FETCH code
new f6eed48 JAMES-3302 Migrate Run section for Distributed server
new 96c4f1d JAMES-3302 Migrate Run with docker section for Distributed server
new 7918cec JAMES-3302 Migrate CLI section for Distributed Server
new 627a3de JAMES-3302 Adapt CLI documentation for the Distributed Server
new 4669739 JAMES-3302 Migrate WebAdmin documentation to Antora
new 1a15f32 JAMES-3302 Refine Distributed server architecture page
new e8937d9 JAMES-3302 Document logging for the Distributed Server
new 7d9bb96 JAMES-3302 Extract architecture from operating guide
new cd105c8 JAMES-3302 Migrate operator guide
new 9d217f1 JAMES-3302 Write more about Architecture for Distributed server
new 4b8e7d7 JAMES-3302 List configuration files for Distributed Server
new 1c9f1d3 JAMES-3302 Fixing dead links for Distributed Server documentation
new 4e099d0 [ADR] Define quality levels
The 31 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/modules/servers/nav.adoc | 1 +
docs/modules/servers/pages/distributed.adoc | 12 +-
.../servers/pages/distributed/architecture.adoc | 228 ++
.../servers/pages/distributed/configure/index.adoc | 22 +-
.../servers/pages/distributed/extend/index.adoc | 8 +-
.../servers/pages/distributed/operate/cli.adoc | 334 +-
.../servers/pages/distributed/operate/guide.adoc | 603 ++-
.../servers/pages/distributed/operate/index.adoc | 10 +-
.../servers/pages/distributed/operate/logging.adoc | 108 +
.../servers/pages/distributed/operate/metrics.adoc | 9 +-
.../pages/distributed/operate/webadmin.adoc | 4047 +++++++++++++++++++-
.../servers/pages/distributed/run-docker.adoc | 120 +-
docs/modules/servers/pages/distributed/run.adoc | 114 +-
.../apache/james/mailbox/model/MessageResult.java | 5 -
.../apache/james/mailbox/MailboxManagerTest.java | 31 -
.../cassandra/mail/CassandraMessageDAO.java | 6 -
.../cassandra/mail/MessageRepresentation.java | 5 +-
.../model/openjpa/AbstractJPAMailboxMessage.java | 6 -
.../mailbox/maildir/mail/model/MaildirMessage.java | 5 -
.../james/mailbox/store/MessageResultImpl.java | 5 -
.../mailbox/store/StoreMessageResultIterator.java | 5 -
.../store/mail/model/DelegatingMailboxMessage.java | 5 -
.../james/mailbox/store/mail/model/Message.java | 2 -
.../mail/model/impl/SimpleMailboxMessage.java | 29 +-
.../store/mail/model/impl/SimpleMessage.java | 9 +-
.../store/AbstractMessageIdManagerStorageTest.java | 32 -
.../metrics/dropwizard/DropWizardTimeMetric.java | 2 +-
pom.xml | 5 +
.../james/imap/message/response/FetchResponse.java | 3 -
.../james/imap/processor/fetch/AddressImpl.java | 4 -
.../imap/processor/fetch/ContentBodyElement.java | 2 -
.../imap/processor/fetch/EnvelopeBuilder.java | 129 +-
.../james/imap/processor/fetch/EnvelopeImpl.java | 11 -
.../imap/processor/fetch/FetchResponseBuilder.java | 94 +-
.../imap/processor/fetch/HeaderBodyElement.java | 1 -
.../imap/processor/fetch/HeadersBodyElement.java | 2 -
.../imap/processor/fetch/MessageResultUtils.java | 78 +-
.../imap/processor/fetch/MimeBodyElement.java | 10 +-
.../processor/fetch/MimeDescriptorStructure.java | 25 +-
.../processor/fetch/PartialFetchBodyElement.java | 15 +-
.../fetch/MailboxMessageResultUtilsTest.java | 10 +-
.../guice/cassandra-rabbitmq-guice/pom.xml | 4 +
.../james/CassandraRabbitMQJamesServerMain.java | 3 +-
server/container/guice/pom.xml | 6 +
.../pom.xml | 8 +-
.../server/RabbitMailQueueRoutesModule.java} | 8 +-
.../RabbitMailQueueTaskSerializationModule.java} | 21 +-
server/container/guice/rabbitmq/pom.xml | 8 +
...dminServerTaskSerializationIntegrationTest.java | 23 +-
server/protocols/webadmin/pom.xml | 1 +
.../protocols/webadmin/webadmin-mailqueue/pom.xml | 2 +-
.../james/webadmin/dto/MailQueueItemDTOTest.java | 3 +-
.../pom.xml | 18 +-
.../webadmin/routes/RabbitMQMailQueuesRoutes.java | 174 +
...rocessedMailsTaskAdditionalInformationDTO.java} | 73 +-
.../service/RepublishNotProcessedMailsTaskDTO.java | 85 +
.../service/RepublishNotprocessedMailsTask.java | 107 +
.../routes/RabbitMQMailQueuesRoutesTest.java | 144 +
.../RepublishNotprocessedMailsTaskTest.java | 108 +
.../james/queue/api/ManageableMailQueue.java | 18 +-
.../james/queue/file/FileCacheableMailQueue.java | 2 +-
.../james/queue/jms/JMSCacheableMailQueue.java | 2 +-
.../james/queue/memory/MemoryMailQueueFactory.java | 4 +-
.../org/apache/james/queue/rabbitmq/Dequeuer.java | 6 +-
.../org/apache/james/queue/rabbitmq/Enqueuer.java | 8 +
.../james/queue/rabbitmq/RabbitMQMailQueue.java | 17 +-
.../queue/rabbitmq/view/api/MailQueueView.java | 9 +-
.../view/cassandra/CassandraMailQueueBrowser.java | 64 +-
.../view/cassandra/CassandraMailQueueView.java | 22 +-
.../queue/rabbitmq/RabbitMQMailQueueTest.java | 193 +-
.../rabbitmq/RabbitMqMailQueueFactoryTest.java | 3 +-
...etedTask.java => FailsDeserializationTask.java} | 6 +-
.../distributed/RabbitMQWorkQueue.java | 19 +-
.../distributed/DistributedTaskManagerTest.java | 122 +-
.../distributed/RabbitMQWorkQueueTest.java | 5 +-
...skDTO.java => FailsDeserializationTaskDTO.java} | 5 +-
.../server/task/json/dto/TestTaskDTOModules.java | 9 +
.../apache/james/utils/DiscreteDistribution.java | 3 +-
.../james/utils/DiscreteDistributionTest.java | 10 +-
src/adr/0040-quality-levels-definitions.md | 62 +
src/site/markdown/server/manage-webadmin.md | 35 +
81 files changed, 7019 insertions(+), 553 deletions(-)
create mode 100644 docs/modules/servers/pages/distributed/operate/logging.adoc
copy server/container/guice/protocols/{webadmin-mailqueue => webadmin-rabbitmq-mailqueue}/pom.xml (85%)
copy server/container/guice/protocols/{webadmin-mailqueue/src/main/java/org/apache/james/modules/server/MailQueueRoutesModule.java => webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueRoutesModule.java} (85%)
copy server/container/guice/protocols/{webadmin-mailbox/src/main/java/org/apache/james/modules/server/WebadminMailboxExportTaskSerializationModule.java => webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueTaskSerializationModule.java} (67%)
copy server/protocols/webadmin/{webadmin-mailqueue => webadmin-rabbitmq}/pom.xml (88%)
create mode 100644 server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutes.java
copy server/protocols/webadmin/{webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/WebAdminClearMailRepositoryTaskAdditionalInformationDTO.java => webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskAdditionalInformationDTO.java} (54%)
create mode 100644 server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskDTO.java
create mode 100644 server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTask.java
create mode 100644 server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutesTest.java
create mode 100644 server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTaskTest.java
copy server/task/task-api/src/test/java/org/apache/james/task/{CompletedTask.java => FailsDeserializationTask.java} (90%)
copy server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/{ThrowingTaskDTO.java => FailsDeserializationTaskDTO.java} (91%)
create mode 100644 src/adr/0040-quality-levels-definitions.md
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 11/31: [REFACTORING] Remove unused methods in
MessageResultUtils
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit d4dd474c4bab4bbc9d23a59386603a039ea69538
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:26:54 2020 +0700
[REFACTORING] Remove unused methods in MessageResultUtils
---
.../imap/processor/fetch/MessageResultUtils.java | 58 ----------------------
.../fetch/MailboxMessageResultUtilsTest.java | 4 +-
2 files changed, 3 insertions(+), 59 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
index 2778747..143dcd8 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
@@ -20,7 +20,6 @@
package org.apache.james.imap.processor.fetch;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
@@ -58,34 +57,6 @@ public class MessageResultUtils {
* @return <code>List</code> of <code>MessageResult.Header</code>'s, in
* their natural order
*/
- public static List<Header> getMatching(String[] names, Iterator<Header> iterator) {
- final List<Header> results = new ArrayList<>(20);
- if (iterator != null) {
- while (iterator.hasNext()) {
- Header header = iterator.next();
- final String headerName = header.getName();
- if (headerName != null) {
- if (Arrays.stream(names)
- .anyMatch(headerName::equalsIgnoreCase)) {
- results.add(header);
- }
- }
- }
- }
- return results;
- }
-
- /**
- * Gets header lines whose header names matches (ignoring case) any of those
- * given.
- *
- * @param names
- * header names to be matched, not null
- * @param iterator
- * {@link Header} <code>Iterator</code>
- * @return <code>List</code> of <code>MessageResult.Header</code>'s, in
- * their natural order
- */
public static List<Header> getMatching(Collection<String> names, Iterator<Header> iterator) {
return matching(names, iterator, false);
}
@@ -153,33 +124,4 @@ public class MessageResultUtils {
}
return result;
}
-
- /**
- * Gets header lines whose header name fails to match (ignoring case) all of
- * the given names.
- *
- * @param names
- * header names, not null
- * @param iterator
- * {@link Header} <code>Iterator</code>
- * @return <code>List</code> of <code>@MessageResult.Header</code>'s, in
- * their natural order
- */
- public static List<Header> getNotMatching(String[] names, Iterator<Header> iterator) {
- final List<Header> results = new ArrayList<>(20);
- if (iterator != null) {
- while (iterator.hasNext()) {
- Header header = iterator.next();
- final String headerName = header.getName();
- if (headerName != null) {
- boolean match = Arrays.stream(names)
- .anyMatch(headerName::equalsIgnoreCase);
- if (!match) {
- results.add(header);
- }
- }
- }
- }
- return results;
- }
}
diff --git a/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java b/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
index e554b51..6a409fa 100644
--- a/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
+++ b/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
@@ -29,9 +29,11 @@ import org.apache.james.mailbox.model.Header;
import org.junit.Before;
import org.junit.Test;
+import com.google.common.collect.ImmutableList;
+
public class MailboxMessageResultUtilsTest {
- private static final String[] NAMES = { "One", "Three" };
+ private static final ImmutableList<String> NAMES = ImmutableList.of("One", "Three");
Header headerOne;
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 09/31: [REFACTORING] FetchResponseBuilder should
use Optional::isEmpty
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit ba3a096ab097e86c30f4db21aa961a5501e5bd25
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:11:08 2020 +0700
[REFACTORING] FetchResponseBuilder should use Optional::isEmpty
---
.../james/imap/processor/fetch/FetchResponseBuilder.java | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
index 4d83116..af19cb5 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
@@ -306,7 +306,7 @@ public final class FetchResponseBuilder {
}
private Content getTextContent(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
- if (!path.isPresent()) {
+ if (path.isEmpty()) {
try {
return messageResult.getBody();
} catch (IOException e) {
@@ -332,7 +332,7 @@ public final class FetchResponseBuilder {
// Check if its base as this can give use a more correctly working check
// to see if we need to write the newline out to the client.
// This is related to IMAP-298
- if (!path.isPresent()) {
+ if (path.isEmpty()) {
if (messageResult.getSize() - result.size() <= 0) {
// Seems like this mail has no body
result.noBody();
@@ -352,7 +352,7 @@ public final class FetchResponseBuilder {
}
private FetchResponse.BodyElement headers(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
- if (!path.isPresent()) {
+ if (path.isEmpty()) {
// if its base we can just return the raw headers without parsing
// them. See MAILBOX-311 and IMAP-?
HeadersBodyElement element = new HeadersBodyElement(name, messageResult.getHeaders());
@@ -388,7 +388,7 @@ public final class FetchResponseBuilder {
}
private Iterator<Header> getHeaders(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
- if (!path.isPresent()) {
+ if (path.isEmpty()) {
return messageResult.getHeaders().headers();
} else {
return messageResult.iterateHeaders(path.get());
@@ -406,7 +406,7 @@ public final class FetchResponseBuilder {
}
private Content getContent(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
- if (!path.isPresent()) {
+ if (path.isEmpty()) {
try {
return messageResult.getFullContent();
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 10/31: [REFACTORING] MessageResultUtils never throws
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 344154778f224971f95ffba2b9a29da6b9622f4c
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:23:51 2020 +0700
[REFACTORING] MessageResultUtils never throws
---
.../imap/processor/fetch/MessageResultUtils.java | 24 +++++++---------------
.../fetch/MailboxMessageResultUtilsTest.java | 6 +++---
2 files changed, 10 insertions(+), 20 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
index 3416670..2778747 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MessageResultUtils.java
@@ -25,9 +25,6 @@ import java.util.Collection;
import java.util.Iterator;
import java.util.List;
-import javax.mail.MessagingException;
-
-import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.model.Header;
public class MessageResultUtils {
@@ -39,8 +36,6 @@ public class MessageResultUtils {
* {@link Header} <code>Iterator</code>
* @return <code>List</code> of <code>MessageResult.Header<code>'s,
* in their natural order
- *
- * @throws MessagingException
*/
public static List<Header> getAll(Iterator<Header> iterator) {
final List<Header> results = new ArrayList<>();
@@ -62,9 +57,8 @@ public class MessageResultUtils {
* {@link Header} <code>Iterator</code>
* @return <code>List</code> of <code>MessageResult.Header</code>'s, in
* their natural order
- * @throws MessagingException
*/
- public static List<Header> getMatching(String[] names, Iterator<Header> iterator) throws MailboxException {
+ public static List<Header> getMatching(String[] names, Iterator<Header> iterator) {
final List<Header> results = new ArrayList<>(20);
if (iterator != null) {
while (iterator.hasNext()) {
@@ -91,13 +85,12 @@ public class MessageResultUtils {
* {@link Header} <code>Iterator</code>
* @return <code>List</code> of <code>MessageResult.Header</code>'s, in
* their natural order
- * @throws MessagingException
*/
- public static List<Header> getMatching(Collection<String> names, Iterator<Header> iterator) throws MailboxException {
+ public static List<Header> getMatching(Collection<String> names, Iterator<Header> iterator) {
return matching(names, iterator, false);
}
- private static List<Header> matching(Collection<String> names, Iterator<Header> iterator, boolean not) throws MailboxException {
+ private static List<Header> matching(Collection<String> names, Iterator<Header> iterator, boolean not) {
final List<Header> results = new ArrayList<>(names.size());
if (iterator != null) {
while (iterator.hasNext()) {
@@ -112,7 +105,7 @@ public class MessageResultUtils {
return results;
}
- private static boolean contains(Collection<String> names, Header header) throws MailboxException {
+ private static boolean contains(Collection<String> names, Header header) {
final String headerName = header.getName();
if (headerName != null) {
return names.stream().anyMatch(name -> name.equalsIgnoreCase(headerName));
@@ -130,9 +123,8 @@ public class MessageResultUtils {
* {@link Header} <code>Iterator</code>
* @return <code>List</code> of <code>MessageResult.Header</code>'s, in
* their natural order
- * @throws MessagingException
*/
- public static List<Header> getNotMatching(Collection<String> names, Iterator<Header> iterator) throws MailboxException {
+ public static List<Header> getNotMatching(Collection<String> names, Iterator<Header> iterator) {
return matching(names, iterator, true);
}
@@ -146,9 +138,8 @@ public class MessageResultUtils {
* not null
* @return <code>MessageResult.Header</code>, or null if the header does not
* exist
- * @throws MessagingException
*/
- public static Header getMatching(String name, Iterator<Header> iterator) throws MailboxException {
+ public static Header getMatching(String name, Iterator<Header> iterator) {
Header result = null;
if (name != null) {
while (iterator.hasNext()) {
@@ -173,9 +164,8 @@ public class MessageResultUtils {
* {@link Header} <code>Iterator</code>
* @return <code>List</code> of <code>@MessageResult.Header</code>'s, in
* their natural order
- * @throws MessagingException
*/
- public static List<Header> getNotMatching(String[] names, Iterator<Header> iterator) throws MailboxException {
+ public static List<Header> getNotMatching(String[] names, Iterator<Header> iterator) {
final List<Header> results = new ArrayList<>(20);
if (iterator != null) {
while (iterator.hasNext()) {
diff --git a/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java b/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
index c77374e..e554b51 100644
--- a/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
+++ b/protocols/imap/src/test/java/org/apache/james/imap/processor/fetch/MailboxMessageResultUtilsTest.java
@@ -62,7 +62,7 @@ public class MailboxMessageResultUtilsTest {
}
@Test
- public void testGetMatching() throws Exception {
+ public void testGetMatching() {
List<Header> results = MessageResultUtils
.getMatching(NAMES, headers.iterator());
assertThat(results.size()).isEqualTo(2);
@@ -71,7 +71,7 @@ public class MailboxMessageResultUtilsTest {
}
@Test
- public void testGetNotMatching() throws Exception {
+ public void testGetNotMatching() {
List<Header> results = MessageResultUtils.getNotMatching(NAMES, headers
.iterator());
assertThat(results.size()).isEqualTo(1);
@@ -79,7 +79,7 @@ public class MailboxMessageResultUtilsTest {
}
@Test
- public void testGetMatchingSingle() throws Exception {
+ public void testGetMatchingSingle() {
assertThat(MessageResultUtils.getMatching("One", headers
.iterator())).isEqualTo(headerOne);
assertThat(MessageResultUtils.getMatching("Three",
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 26/31: JAMES-3302 Extract architecture from
operating guide
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7d9bb9676a201e3c86d46eb0d4cc3a4818262444
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 14:24:49 2020 +0700
JAMES-3302 Extract architecture from operating guide
---
.../servers/pages/distributed/architecture.adoc | 145 ++++++++++++++++++++-
1 file changed, 144 insertions(+), 1 deletion(-)
diff --git a/docs/modules/servers/pages/distributed/architecture.adoc b/docs/modules/servers/pages/distributed/architecture.adoc
index 3113be9..d4e1678 100644
--- a/docs/modules/servers/pages/distributed/architecture.adoc
+++ b/docs/modules/servers/pages/distributed/architecture.adoc
@@ -19,4 +19,147 @@ In order to deliver its promises, the Distributed Server leverages the following
== Components
-(TODO)
\ No newline at end of file
+This section presents the various components of the Distributed server, providing context about
+their interactions, and about their implementations.
+
+=== Mail processing
+
+Mail processing allows to take asynchronously business decisions on
+received emails.
+
+Here are its components:
+
+* The `spooler` takes mail out of the mailQueue and executes mail
+processing within the `mailet container`.
+* The `mailet container` synchronously executes the user defined logic.
+This `logic' is written through the use of `mailet`, `matcher` and
+`processor`.
+* A `mailet` represents an action: mail modification, envelop
+modification, a side effect, or stop processing.
+* A `matcher` represents a condition to execute a mailet.
+* A `processor` is a flow of pair of `matcher` and `mailet` executed
+sequentially. The `ToProcessor` mailet is a `goto` instruction to start
+executing another `processor`
+* A `mail repository` allows storage of a mail as part of its
+processing. Standard configuration relies on the following mail
+repository:
+** `cassandra://var/mail/error/` : unexpected errors that occurred
+during mail processing. Emails impacted by performance related
+exceptions, or logical bug within James code are typically stored here.
+These mails could be reprocessed once the cause of the error is fixed.
+The `Mail.error` field can help diagnose the issue. Correlation with
+logs can be achieved via the use of the `Mail.name` field.
+** `cassandra://var/mail/address-error/` : mail addressed to a
+non-existing recipient of a handled local domain. These mails could be
+reprocessed once the user is created, for instance.
+** `cassandra://var/mail/relay-denied/` : mail for whom relay was
+denied: missing authentication can, for instance, be a cause. In
+addition to prevent disasters upon miss configuration, an email review
+of this mail repository can help refine a host spammer blacklist.
+** `cassandra://var/mail/rrt-error/` : runtime error upon Recipient
+Rewritting occurred. This is typically due to a loop.
+
+=== Mail Queue
+
+An email queue is a mandatory component of SMTP servers. It is a system
+that creates a queue of emails that are waiting to be processed for
+delivery. Email queuing is a form of Message Queuing – an asynchronous
+service-to-service communication. A message queue is meant to decouple a
+producing process from a consuming one. An email queue decouples email
+reception from email processing. It allows them to communicate without
+being connected. As such, the queued emails wait for processing until
+the recipient is available to receive them. As James is an Email Server,
+it also supports mail queue as well.
+
+==== Why Mail Queue is necessary
+
+You might often need to check mail queue to make sure all emails are
+delivered properly. At first, you need to know why email queues get
+clogged. Here are the two core reasons for that:
+
+* Exceeded volume of emails
+
+Some mailbox providers enforce email rate limits on IP addresses. The
+limits are based on the sender reputation. If you exceeded this rate and
+queued too many emails, the delivery speed will decrease.
+
+* Spam-related issues
+
+Another common reason is that your email has been busted by spam
+filters. The filters will let the emails gradually pass to analyze how
+the rest of the recipients react to the message. If there is slow
+progress, it’s okay. Your email campaign is being observed and assessed.
+If it’s stuck, there could be different reasons including the blockage
+of your IP address.
+
+==== Why combining Cassandra, RabbitMQ and Object storage for MailQueue
+
+* RabbitMQ ensures the messaging function, and avoids polling.
+* Cassandra enables administrative operations such as browsing, deleting
+using a time series which might require fine performance tuning (see
+http://cassandra.apache.org/doc/latest/operating/index.html[Operating
+Casandra documentation]).
+* Object Storage stores potentially large binary payload.
+
+However the current design do not implement delays. Delays allow to
+define the time a mail have to be living in the mailqueue before being
+dequeued and is used for example for exponential wait delays upon remote
+delivery retries, or
+
+=== Mailbox
+
+(TODO)
+
+==== Event Bus
+
+Distributed James relies on an event bus system to enrich mailbox capabilities. Each
+operation performed on the mailbox will trigger related events, that can
+be processed asynchronously by potentially any James node on a
+distributed system.
+
+Many different kind of events can be triggered during a mailbox
+operation, such as:
+
+* `MailboxEvent`: event related to an operation regarding a mailbox:
+** `MailboxDeletion`: a mailbox has been deleted
+** `MailboxAdded`: a mailbox has been added
+** `MailboxRenamed`: a mailbox has been renamed
+** `MailboxACLUpdated`: a mailbox got its rights and permissions updated
+* `MessageEvent`: event related to an operation regarding a message:
+** `Added`: messages have been added to a mailbox
+** `Expunged`: messages have been expunged from a mailbox
+** `FlagsUpdated`: messages had their flags updated
+** `MessageMoveEvent`: messages have been moved from a mailbox to an
+other
+* `QuotaUsageUpdatedEvent`: event related to quota update
+
+Mailbox listeners can register themselves on this event bus system to be
+called when an event is fired, allowing to do different kind of extra
+operations on the system, like:
+
+* Current quota calculation
+* Message indexation with ElasticSearch
+* Mailbox annotations cleanup
+* Ham/spam reporting to SpamAssassin
+* …
+
+==== Deleted Messages Vault
+
+Deleted Messages Vault is an interesting feature that will help James
+users have a chance to:
+
+* retain users deleted messages for some time.
+* restore & export deleted messages by various criteria.
+* permanently delete some retained messages.
+
+If the Deleted Messages Vault is enabled when users delete their mails,
+and by that we mean when they try to definitely delete them by emptying
+the trash, James will retain these mails into the Deleted Messages
+Vault, before an email or a mailbox is going to be deleted. And only
+administrators can interact with this component via
+wref:webadmin.adoc#_deleted-messages-vault[WebAdmin REST APIs].
+
+However, mails are not retained forever as you have to configure a
+retention period before using it (with one-year retention by default if
+not defined). It’s also possible to permanently delete a mail if needed.
+
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 19/31: JAMES-3302 Migrate Run section for
Distributed server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f6eed48db45682a76bd17a9e8ed4e75ab073af12
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 11:49:25 2020 +0700
JAMES-3302 Migrate Run section for Distributed server
---
docs/modules/servers/pages/distributed/run.adoc | 114 +++++++++++++++++++++++-
1 file changed, 112 insertions(+), 2 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/run.adoc b/docs/modules/servers/pages/distributed/run.adoc
index d04f912..4baacdc 100644
--- a/docs/modules/servers/pages/distributed/run.adoc
+++ b/docs/modules/servers/pages/distributed/run.adoc
@@ -1,4 +1,114 @@
= Run
-(TODO adapt content from
-https://github.com/linagora/james-project/blob/master/src/site/markdown/server/install/guice-cassandra-rabbitmq-swift.md)
\ No newline at end of file
+== Building
+
+=== Requirements
+
+* Java 11 SDK
+* Docker ∕ ElasticSearch 6.3.2, RabbitMQ Management 3.3.7, Swift
+ObjectStorage 2.15.1 and Cassandra 3.11.3
+* Maven 3
+
+=== Building the artifacts
+
+An usual compilation using maven will produce two artifacts into
+server/container/guice/cassandra-rabbitmq-guice/target directory:
+
+* james-server-cassandra-rabbitmq-guice.jar
+* james-server-cassandra-rabbitmq-guice.lib
+
+You can for example run in the base of
+https://github.com/apache/james-project[this git repository]:
+
+....
+mvn clean install
+....
+
+== Running
+
+=== Requirements
+
+* Cassandra 3.11.3
+* ElasticSearch 6.3.2
+* RabbitMQ-Management 3.8.1
+* Swift ObjectStorage 2.15.1 or Scality S3 server or AWS S3
+
+=== James Launch
+
+To run james, you have to create a directory containing required
+configuration files.
+
+James requires the configuration to be in a subfolder of working
+directory that is called *conf*. You can get a sample directory for
+configuration from
+https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf[dockerfiles/run/guice/cassandra-rabbitmq/destination/conf].
+You might need to adapt it to your needs.
+
+You also need to generate a keystore in your conf folder with the
+following command:
+
+[source,bash]
+----
+$ keytool -genkey -alias james -keyalg RSA -keystore conf/keystore
+----
+
+You need to have a Cassandra, ElasticSearch and RabbitMQ instance
+running. You can either install the servers or launch them via docker:
+
+[source,bash]
+----
+$ docker run -d -p 9042:9042 --name=cassandra cassandra:3.11.3
+$ docker run -d -p 9200:9200 --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:6.3.2
+$ docker run -d -p 5672:5672 -p 15672:15672 --name=rabbitmq rabbitmq:3.8.1-management
+$ docker run -d -p 5000:5000 -p 8080:8080 -p 35357:35357 --name=swift linagora/openstack-keystone-swift:pike
+----
+
+Once everything is set up, you just have to run the jar with:
+
+[source,bash]
+----
+$ java -Dworking.directory=. -jar target/james-server-cassandra-rabbitmq-guice.jar
+----
+
+==== Using AWS S3 of Scality S3 server
+
+In order to use AWS S3 or a compatible implementation,
+`blobstore.propeties` has to be filled with:
+
+....
+objectstorage.provider=aws-s3
+objectstorage.namespace=james
+objectstorage.s3.endPoint=http://scality:8080/
+objectstorage.s3.accessKeyId=accessKey1
+objectstorage.s3.secretKey=verySecretKey1
+....
+
+To use Scality S3 server you have to launch it instead of swift
+container:
+
+....
+$ docker run -d -p 8080:8000 --name=s3 scality/s3server:6018536a
+....
+
+More information about available options
+https://hub.docker.com/r/scality/s3server[here].
+
+== Guice-cassandra-rabbitmq-ldap
+
+You can follow the same guide to build and run
+guice-cassandra-rabbitmq-swift-ldap artifact, except that:
+
+ * The *jar* and *libs* needs to be retrieve from
+server/container/guice/cassandra-rabbitmq-ldap-guice/target after
+compilation
+ * The sample configuration can be found in
+https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf[dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf]
+ * You need to configure James to be connecting to a running LDAP server.
+The configuration file is located in
+https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/usersrepository.xml[dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/usersrepository.xml]
+ * You can then launch James via this command:
+
+[source,bash]
+----
+$ java -Dworking.directory=. -jar target/james-server-cassandra-rabbitmq-ldap-guice.jar
+----
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 17/31: [REFACTORING] Rearrange IMAP FETCH fields
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 83b4d2bd4e9479cf45d4e57d0cc242c6ad716c8f
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:45:21 2020 +0700
[REFACTORING] Rearrange IMAP FETCH fields
- Avoid skipping lines between fields
- Use `this` prefix everywhere in constructors
- Remove constructor Object super call that could be implicit
---
.../james/imap/processor/fetch/AddressImpl.java | 4 ----
.../imap/processor/fetch/ContentBodyElement.java | 2 --
.../james/imap/processor/fetch/EnvelopeImpl.java | 11 ----------
.../imap/processor/fetch/FetchResponseBuilder.java | 11 ----------
.../imap/processor/fetch/HeaderBodyElement.java | 1 -
.../imap/processor/fetch/HeadersBodyElement.java | 1 -
.../imap/processor/fetch/MimeBodyElement.java | 4 ----
.../processor/fetch/MimeDescriptorStructure.java | 25 ++++++----------------
.../processor/fetch/PartialFetchBodyElement.java | 7 +-----
9 files changed, 8 insertions(+), 58 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/AddressImpl.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/AddressImpl.java
index 018d519..ffb63fe 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/AddressImpl.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/AddressImpl.java
@@ -26,15 +26,11 @@ import org.apache.james.imap.message.response.FetchResponse;
final class AddressImpl implements FetchResponse.Envelope.Address {
private final String atDomainList;
-
private final String hostName;
-
private final String mailboxName;
-
private final String personalName;
public AddressImpl(String atDomainList, String hostName, String mailboxName, String personalName) {
- super();
this.atDomainList = atDomainList;
this.hostName = hostName;
this.mailboxName = mailboxName;
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/ContentBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/ContentBodyElement.java
index 93dda65..5cbf722 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/ContentBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/ContentBodyElement.java
@@ -31,11 +31,9 @@ import org.apache.james.mailbox.model.Content;
class ContentBodyElement implements BodyElement {
private final String name;
-
protected final Content content;
public ContentBodyElement(String name, Content content) {
- super();
this.name = name;
this.content = content;
}
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeImpl.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeImpl.java
index 615558f..cfa2f20 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeImpl.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeImpl.java
@@ -25,29 +25,18 @@ package org.apache.james.imap.processor.fetch;
import org.apache.james.imap.message.response.FetchResponse;
final class EnvelopeImpl implements FetchResponse.Envelope {
-
private final Address[] bcc;
-
private final Address[] cc;
-
private final String date;
-
private final Address[] from;
-
private final String inReplyTo;
-
private final String messageId;
-
private final Address[] replyTo;
-
private final Address[] sender;
-
private final String subject;
-
private final Address[] to;
public EnvelopeImpl(String date, String subject, Address[] from, Address[] sender, Address[] replyTo, Address[] to, Address[] cc, Address[] bcc, String inReplyTo, String messageId) {
- super();
this.bcc = bcc;
this.cc = cc;
this.date = date;
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
index af19cb5..0156e7f 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
@@ -55,31 +55,20 @@ import org.apache.james.mailbox.model.MessageResult;
import org.apache.james.mailbox.model.MimePath;
public final class FetchResponseBuilder {
-
private final EnvelopeBuilder envelopeBuilder;
private MessageSequenceNumber msn;
-
private MessageUid uid;
-
private Flags flags;
-
private Date internalDate;
-
private Long size;
-
private ModSeq modSeq;
-
private List<FetchResponse.BodyElement> elements;
-
private FetchResponse.Envelope envelope;
-
private FetchResponse.Structure body;
-
private FetchResponse.Structure bodystructure;
public FetchResponseBuilder(EnvelopeBuilder envelopeBuilder) {
- super();
this.envelopeBuilder = envelopeBuilder;
}
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeaderBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeaderBodyElement.java
index e2b3abd..d7656ce 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeaderBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeaderBodyElement.java
@@ -37,7 +37,6 @@ public class HeaderBodyElement extends MimeBodyElement {
public HeaderBodyElement(String name, List<Header> headers) throws MailboxException {
super(name, headers);
}
-
/**
* Indicate that there is no text body in the message. In this case we don't need to write a single CRLF in anycase if
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
index 9b81c2e..fcdcb58 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
@@ -31,7 +31,6 @@ public class HeadersBodyElement extends ContentBodyElement {
super(name, content);
}
-
/**
* Indicate that there is no text body in the message. In this case we don't need to write a single CRLF in anycase if
* this Element does not contain a header.
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
index 715674c..7a46285 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
@@ -37,17 +37,13 @@ import org.apache.james.mailbox.model.Header;
*/
public class MimeBodyElement implements BodyElement {
private final String name;
-
protected final List<Header> headers;
-
protected long size;
public MimeBodyElement(String name, List<Header> headers) throws MailboxException {
- super();
this.name = name;
this.headers = headers;
this.size = calculateSize(headers);
-
}
@Override
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeDescriptorStructure.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeDescriptorStructure.java
index 59c01ae..0b44c99 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeDescriptorStructure.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeDescriptorStructure.java
@@ -34,34 +34,23 @@ import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.model.MimeDescriptor;
final class MimeDescriptorStructure implements FetchResponse.Structure {
-
private final MimeDescriptor descriptor;
-
private final List<String> parameters;
-
private final List<Structure> parts;
-
private final String disposition;
-
private final Map<String, String> dispositionParams;
-
private final String location;
-
private final String md5;
-
private final List<String> languages;
-
private final Structure embeddedMessageStructure;
-
private final Envelope envelope;
public MimeDescriptorStructure(boolean allowExtensions, MimeDescriptor descriptor, EnvelopeBuilder builder) throws MailboxException {
- super();
this.descriptor = descriptor;
- parameters = createParameters(descriptor);
- parts = createParts(allowExtensions, descriptor, builder);
+ this.parameters = createParameters(descriptor);
+ this.parts = createParts(allowExtensions, descriptor, builder);
- languages = descriptor.getLanguages();
+ this.languages = descriptor.getLanguages();
this.dispositionParams = descriptor.getDispositionParams();
this.disposition = descriptor.getDisposition();
@@ -70,11 +59,11 @@ final class MimeDescriptorStructure implements FetchResponse.Structure {
final MimeDescriptor embeddedMessage = descriptor.embeddedMessage();
if (embeddedMessage == null) {
- embeddedMessageStructure = null;
- envelope = null;
+ this.embeddedMessageStructure = null;
+ this.envelope = null;
} else {
- embeddedMessageStructure = new MimeDescriptorStructure(allowExtensions, embeddedMessage, builder);
- envelope = builder.buildEnvelope(embeddedMessage);
+ this.embeddedMessageStructure = new MimeDescriptorStructure(allowExtensions, embeddedMessage, builder);
+ this.envelope = builder.buildEnvelope(embeddedMessage);
}
}
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
index 78b3f8d..f2546cc 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
@@ -29,21 +29,16 @@ import org.apache.james.imap.message.response.FetchResponse.BodyElement;
* Wraps full content to implement a partial fetch.
*/
final class PartialFetchBodyElement implements BodyElement {
-
private final BodyElement delegate;
-
private final long firstOctet;
-
private final long numberOfOctets;
-
private final String name;
public PartialFetchBodyElement(BodyElement delegate, long firstOctet, long numberOfOctets) {
- super();
this.delegate = delegate;
this.firstOctet = firstOctet;
this.numberOfOctets = numberOfOctets;
- name = delegate.getName() + "<" + firstOctet + ">";
+ this.name = delegate.getName() + "<" + firstOctet + ">";
}
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 22/31: JAMES-3302 Adapt CLI documentation for the
Distributed Server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 627a3de9e2f8afce033caeda7f5f1c454b18e44a
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 12:04:13 2020 +0700
JAMES-3302 Adapt CLI documentation for the Distributed Server
And fix typos
---
.../servers/pages/distributed/operate/cli.adoc | 24 ++++++++--------------
1 file changed, 9 insertions(+), 15 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/operate/cli.adoc b/docs/modules/servers/pages/distributed/operate/cli.adoc
index 2b90094..4e8367d 100644
--- a/docs/modules/servers/pages/distributed/operate/cli.adoc
+++ b/docs/modules/servers/pages/distributed/operate/cli.adoc
@@ -1,14 +1,8 @@
= Command Line Interface
-With any wiring, James is packed with a command line client.
+The distributed server is packed with a command line client.
-To use it enter, for Spring distrubution:
-
-....
-./bin/james-cli.sh -h 127.0.0.1 -p 9999 COMMAND
-....
-
-And for Guice distributions:
+To run this command line client simply execute:
....
java -jar /root/james-cli.jar -h 127.0.0.1 -p 9999 COMMAND
@@ -17,7 +11,7 @@ java -jar /root/james-cli.jar -h 127.0.0.1 -p 9999 COMMAND
The following document will explain you which are the available options
for *COMMAND*.
-Note: the command line before *COMMAND* will be documente as _\{cli}_.
+Note: the above command line before *COMMAND* will be documented as _\{cli}_.
== Manage Domains
@@ -173,8 +167,8 @@ INBOX.subFolder mailbox belonging to user user@domain.tld.
== Managing mappings
-A mapping is a recipient rewritting rule. There is several kind of
-rewritting rules:
+A mapping is a recipient rewriting rule. There is several kind of
+rewriting rules:
* address mapping: rewrite a given mail address into an other one.
* regex mapping.
@@ -283,7 +277,7 @@ level. Note: syntax is similar to what was exposed previously.
== Re-indexing
James allow you to index your emails in a search engine, for making
-search faster. Both ElasticSearch and Lucene are supported.
+search faster.
For some reasons, you might want to re-index your mails (inconsistencies
across datastore, migrations).
@@ -294,7 +288,7 @@ To re-index all mails of all mailboxes of all users, type:
{cli} ReindexAll
....
-And for a precise mailbox:
+And for a specific mailbox:
....
{cli} Reindex #private user@domain.tld INBOX
@@ -302,7 +296,7 @@ And for a precise mailbox:
== Sieve scripts quota
-James implements Sieve (RFC-5228). Your users can then writte scripts
+James implements Sieve (RFC-5228). Your users can then write scripts
and upload them to the server. Thus they can define the desired behavior
upon email reception. James defines a Sieve mailet for this, and stores
Sieve scripts. You can update them via the ManageSieve protocol, or via
@@ -331,7 +325,7 @@ And for specific user quotas:
Migration is experimental for now. You would need to customize *Spring*
configuration to add a new mailbox manager with a different bean name.
-You can then copy data accross mailbox managers using:
+You can then copy data across mailbox managers using:
....
{cli} CopyMailbox srcBean dstBean
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 29/31: JAMES-3302 List configuration files for
Distributed Server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4b8e7d7843c20a11747bad53d13421a372ddd7b9
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 18:08:33 2020 +0700
JAMES-3302 List configuration files for Distributed Server
---
.../servers/pages/distributed/configure/index.adoc | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/docs/modules/servers/pages/distributed/configure/index.adoc b/docs/modules/servers/pages/distributed/configure/index.adoc
index dae4ee1..f1c16da 100644
--- a/docs/modules/servers/pages/distributed/configure/index.adoc
+++ b/docs/modules/servers/pages/distributed/configure/index.adoc
@@ -4,4 +4,24 @@ This section presents how to configure the Distributed server.
The following configuration files are exposed:
-(TODO)
\ No newline at end of file
+* *batchsizes.properties* allows to configure mailbox read batch sizes link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/batchsizes.properties[example]
+* *blobstore.properties* allows to configure the BlobStore link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/blob.properties[example]
+* *cassandra.properties* allows to configure the Cassandra driver link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties[example]
+* *deletedMessageVault.properties* allows to configure the DeletedMessageVaultlink:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/deletedMessageVault.properties[example]
+* *dnsservice.xml* allows to configure DNS resolution link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/dnsservice.xml[example]
+* *domainlist.xml* allows to configure Domain storage link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/domainlist.xml[example]
+* *elasticsearch.properties* allows to configure ElasticSearch driverlink:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties[example]
+* *extensions.properties* allows to extend James behaviour by loading yours extensions in it link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/extensions.properties[example]
+* *healthcheck.properties* allows to configure periodical healthchecks link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/healthcheck.properties[example]
+* *jmap.properties* allows to configure the JMAP protocol link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/jmap.properties[example]
+* *jmx.properties* allows configuration of JMX being use by the Command Line Interface link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/jmx.properties[example]
+* *listeners.xml* enables configuration of Mailbox Listeners link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/listeners.xml[example]
+* *lmtpserver.xml* allows configuring the LMTP protocol link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/lmtpserver.xml[example]
+* *mailetcontainer.xml* allows configuring mail processing link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml[example]
+* *mailrepositorystore.xml* enables registration of allowed MailRepository protcols and link them to MailRepository implementations link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailrepositorystore.xml[example]
+* *managesieveserver.xml* allows configuration for ManagedSieve (unsupported) link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/managesieveserver.xml[example]
+* *pop3server.xml* allows configuration for the POP3 protocol (experimental) link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/pop3server.xml[example]
+* *rabbitmq.propertiesl* allows configuration for the RabbitMQ driver link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/rabbitmq.properties[example]
+* *tika.properties* allows configuring Tika as a backend for text extraction link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/tika.properties[example]
+* *usersrepository.xml* allowslink:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/usersrepository.xml[example]
+* *webadmin.properties* enables configuration for the WebAdmin protocol link:https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/webadmin.properties[example]
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 21/31: JAMES-3302 Migrate CLI section for
Distributed Server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7918cec82fe7b57d6b9c3324842e2d2883a9315f
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 11:58:26 2020 +0700
JAMES-3302 Migrate CLI section for Distributed Server
---
.../servers/pages/distributed/operate/cli.adoc | 340 ++++++++++++++++++++-
1 file changed, 338 insertions(+), 2 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/operate/cli.adoc b/docs/modules/servers/pages/distributed/operate/cli.adoc
index 35d6fee..2b90094 100644
--- a/docs/modules/servers/pages/distributed/operate/cli.adoc
+++ b/docs/modules/servers/pages/distributed/operate/cli.adoc
@@ -1,4 +1,340 @@
= Command Line Interface
-(TODO migrate and adapt content from
-https://github.com/linagora/james-project/blob/master/src/site/markdown/server/manage-cli.md)
\ No newline at end of file
+With any wiring, James is packed with a command line client.
+
+To use it enter, for Spring distrubution:
+
+....
+./bin/james-cli.sh -h 127.0.0.1 -p 9999 COMMAND
+....
+
+And for Guice distributions:
+
+....
+java -jar /root/james-cli.jar -h 127.0.0.1 -p 9999 COMMAND
+....
+
+The following document will explain you which are the available options
+for *COMMAND*.
+
+Note: the command line before *COMMAND* will be documente as _\{cli}_.
+
+== Manage Domains
+
+Domains represent the domain names handled by your server.
+
+You can add a domain:
+
+....
+{cli} AddDomain domain.tld
+....
+
+You can remove a domain:
+
+....
+{cli} RemoveDomain domain.tld
+....
+
+(Note: associated users are not removed automatically)
+
+Check if a domain is handled:
+
+....
+{cli} ContainsDomain domain.tld
+....
+
+And list your domains:
+
+....
+{cli} ListDomains
+....
+
+== Managing users
+
+Note: the following commands are explained with virtual hosting turned
+on.
+
+Users are accounts on the mail server. James can maintain mailboxes for
+them.
+
+You can add a user:
+
+....
+{cli} AddUser user@domain.tld password
+....
+
+Note: the domain used should have been previously created.
+
+You can delete a user:
+
+....
+{cli} RemoveUser user@domain.tld
+....
+
+(Note: associated mailboxes are not removed automatically)
+
+And change a user password:
+
+....
+{cli} SetPassword user@domain.tld password
+....
+
+Note: All these write operations can not be performed on LDAP backend,
+as the implementation is read-only.
+
+Finally, you can list users:
+
+....
+{cli} ListUsers
+....
+
+=== Virtual hosting
+
+James supports virtualhosting.
+
+* If set to true in the configuration, then the username is the full
+mail address.
+
+The domains then become a part of the user.
+
+_usera@domaina.com and_ _usera@domainb.com_ on a mail server with
+_domaina.com_ and _domainb.com_ configured are mail addresses that
+belongs to different users.
+
+* If set to false in the configurations, then the username is the mail
+address local part.
+
+It means that a user is automatically created for all the domains
+configured on your server.
+
+_usera@domaina.com and_ _usera@domainb.com_ on a mail server with
+_domaina.com_ and _domainb.com_ configured are mail addresses that
+belongs to the same users.
+
+Here are some sample commands for managing users when virtual hosting is
+turned off:
+
+....
+{cli} AddUser user password
+{cli} RemoveUser user
+{cli} SetPassword user password
+....
+
+== Managing mailboxes
+
+An administrator can perform some basic operation on user mailboxes.
+
+Note on mailbox formatting: mailboxes are composed of three parts.
+
+* The namespace, indicating what kind of mailbox it is. (Shared or
+not?). The value for users mailboxes is #private . Note that for now no
+other values are supported as James do not support shared mailboxes.
+* The username as stated above, depending on the virtual hosting value.
+* And finally mailbox name. Be aware that `.' serves as mailbox
+hierarchy delimiter.
+
+An administrator can delete all of the mailboxes of a user, which is not
+done automatically when removing a user (to avoid data loss):
+
+....
+{cli} DeleteUserMailboxes user@domain.tld
+....
+
+He can delete a specific mailbox:
+
+....
+{cli} DeleteMailbox #private user@domain.tld INBOX.toBeDeleted
+....
+
+He can list the mailboxes of a specific user:
+
+....
+{cli} ListUserMailboxes user@domain.tld
+....
+
+And finally can create a specific mailbox:
+
+....
+{cli} CreateMailbox #private user@domain.tld INBOX.newFolder
+....
+
+== Adding a message in a mailbox
+
+The administrator can use the CLI to add a message in a mailbox. this
+can be done using:
+
+....
+{cli} ImportEml #private user@domain.tld INBOX.newFolder /full/path/to/file.eml
+....
+
+This command will add a message having the content specified in file.eml
+(that needs to be at the EML format). It will get added in the
+INBOX.subFolder mailbox belonging to user user@domain.tld.
+
+== Managing mappings
+
+A mapping is a recipient rewritting rule. There is several kind of
+rewritting rules:
+
+* address mapping: rewrite a given mail address into an other one.
+* regex mapping.
+
+You can manage address mapping like (redirects email from
+fromUser@fromDomain.tld to redirected@domain.new, then deletes the
+mapping):
+
+....
+{cli} AddAddressMapping fromUser fromDomain.tld redirected@domain.new
+{cli} RemoveAddressMapping fromUser fromDomain.tld redirected@domain.new
+....
+
+You can manage regex mapping like this:
+
+....
+{cli} AddRegexMapping redirected domain.new .*@domain.tld
+{cli} RemoveRegexMapping redirected domain.new .*@domain.tld
+....
+
+You can view mapping for a mail address:
+
+....
+{cli} ListUserDomainMappings user domain.tld
+....
+
+And all mappings defined on the server:
+
+....
+{cli} ListMappings
+....
+
+== Manage quotas
+
+Quotas are limitations on a group of mailboxes. They can limit the
+*size* or the *messages count* in a group of mailboxes.
+
+James groups by defaults mailboxes by user (but it can be overridden),
+and labels each group with a quotaroot.
+
+To get the quotaroot a given mailbox belongs to:
+
+....
+{cli} GetQuotaroot #private user@domain.tld INBOX
+....
+
+Then you can get the specific quotaroot limitations.
+
+For the number of messages:
+
+....
+{cli} GetMessageCountQuota quotaroot
+....
+
+And for the storage space available:
+
+....
+{cli} GetStorageQuota quotaroot
+....
+
+You see the maximum allowed for these values:
+
+For the number of messages:
+
+....
+{cli} GetMaxMessageCountQuota quotaroot
+....
+
+And for the storage space available:
+
+....
+{cli} GetMaxStorageQuota quotaroot
+....
+
+You can also specify maximum for these values.
+
+For the number of messages:
+
+....
+{cli} SetMaxMessageCountQuota quotaroot value
+....
+
+And for the storage space available:
+
+....
+{cli} SetMaxStorageQuota quotaroot value
+....
+
+With value being an integer. Please note the use of units for storage
+(K, M, G). For instance:
+
+....
+{cli} SetMaxStorageQuota someone@apache.org 4G
+....
+
+Moreover, James allows to specify global maximum values, at the server
+level. Note: syntax is similar to what was exposed previously.
+
+....
+{cli} SetGlobalMaxMessageCountQuota value
+{cli} GetGlobalMaxMessageCountQuota
+{cli} SetGlobalMaxStorageQuota value
+{cli} GetGlobalMaxStorageQuota
+....
+
+== Re-indexing
+
+James allow you to index your emails in a search engine, for making
+search faster. Both ElasticSearch and Lucene are supported.
+
+For some reasons, you might want to re-index your mails (inconsistencies
+across datastore, migrations).
+
+To re-index all mails of all mailboxes of all users, type:
+
+....
+{cli} ReindexAll
+....
+
+And for a precise mailbox:
+
+....
+{cli} Reindex #private user@domain.tld INBOX
+....
+
+== Sieve scripts quota
+
+James implements Sieve (RFC-5228). Your users can then writte scripts
+and upload them to the server. Thus they can define the desired behavior
+upon email reception. James defines a Sieve mailet for this, and stores
+Sieve scripts. You can update them via the ManageSieve protocol, or via
+the ManageSieveMailet.
+
+You can define quota for the total size of Sieve scripts, per user.
+
+Syntax is similar to what was exposed for quotas. For defaults values:
+
+....
+{cli} GetSieveQuota
+{cli} SetSieveQuota value
+{cli} RemoveSieveQuota
+....
+
+And for specific user quotas:
+
+....
+{cli} GetSieveUserQuota user@domain.tld
+{cli} SetSieveQuota user@domain.tld value
+{cli} RemoveSieveUserQuota user@domain.tld
+....
+
+== Switching of mailbox implementation
+
+Migration is experimental for now. You would need to customize *Spring*
+configuration to add a new mailbox manager with a different bean name.
+
+You can then copy data accross mailbox managers using:
+
+....
+{cli} CopyMailbox srcBean dstBean
+....
+
+You will then need to reconfigure James to use the new mailbox manager.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 18/31: [REFACTORING] Remove uneeded else blocks in
IMAP FETCH code
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit a78237ba2542c02ee4c2108f0e2a288b660da1ed
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 15:50:05 2020 +0700
[REFACTORING] Remove uneeded else blocks in IMAP FETCH code
---
.../imap/processor/fetch/EnvelopeBuilder.java | 91 ++++++++++------------
.../imap/processor/fetch/FetchResponseBuilder.java | 26 +++----
2 files changed, 50 insertions(+), 67 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
index 7513443..1b2595c 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
@@ -65,33 +65,28 @@ public final class EnvelopeBuilder {
final Header header = MessageResultUtils.getMatching(headerName, message.headers());
if (header == null) {
return null;
- } else {
- final String value = header.getValue();
- if (value == null || "".equals(value)) {
- return null;
- } else {
-
- // ENVELOPE header values must be unfolded
- // See IMAP-269
- //
- //
- // IMAP-Servers are advised to also replace tabs with single spaces while doing the unfolding. This is what javamails
- // unfold does. mime4j's unfold does strictly follow the rfc and so preserve them
- //
- // See IMAP-327 and https://mailman2.u.washington.edu/mailman/htdig/imap-protocol/2010-July/001271.html
- return MimeUtility.unfold(value);
-
- }
}
+ final String value = header.getValue();
+ if (value == null || "".equals(value)) {
+ return null;
+ }
+ // ENVELOPE header values must be unfolded
+ // See IMAP-269
+ //
+ //
+ // IMAP-Servers are advised to also replace tabs with single spaces while doing the unfolding. This is what javamails
+ // unfold does. mime4j's unfold does strictly follow the rfc and so preserve them
+ //
+ // See IMAP-327 and https://mailman2.u.washington.edu/mailman/htdig/imap-protocol/2010-July/001271.html
+ return MimeUtility.unfold(value);
}
private FetchResponse.Envelope.Address[] buildAddresses(Headers message, String headerName, FetchResponse.Envelope.Address[] defaults) throws MailboxException {
final FetchResponse.Envelope.Address[] addresses = buildAddresses(message, headerName);
if (addresses == null) {
return defaults;
- } else {
- return addresses;
}
+ return addresses;
}
/**
@@ -104,41 +99,37 @@ public final class EnvelopeBuilder {
final Header header = MessageResultUtils.getMatching(headerName, message.headers());
if (header == null) {
return null;
- } else {
+ }
+ // We need to unfold the header line.
+ // See https://issues.apache.org/jira/browse/IMAP-154
+ //
+ // IMAP-Servers are advised to also replace tabs with single spaces while doing the unfolding. This is what javamails
+ // unfold does. mime4j's unfold does strictly follow the rfc and so preserve them
+ //
+ // See IMAP-327 and https://mailman2.u.washington.edu/mailman/htdig/imap-protocol/2010-July/001271.html
+ String value = MimeUtility.unfold(header.getValue());
+ if ("".equals(value.trim())) {
+ return null;
+ }
+ AddressList addressList = LenientAddressParser.DEFAULT.parseAddressList(value);
+ final int size = addressList.size();
+ final List<FetchResponse.Envelope.Address> addresses = new ArrayList<>(size);
+ for (Address address : addressList) {
+ if (address instanceof Group) {
+ final Group group = (Group) address;
+ addAddresses(group, addresses);
+
+ } else if (address instanceof Mailbox) {
+ final Mailbox mailbox = (Mailbox) address;
+ final FetchResponse.Envelope.Address mailboxAddress = buildMailboxAddress(mailbox);
+ addresses.add(mailboxAddress);
- // We need to unfold the header line.
- // See https://issues.apache.org/jira/browse/IMAP-154
- //
- // IMAP-Servers are advised to also replace tabs with single spaces while doing the unfolding. This is what javamails
- // unfold does. mime4j's unfold does strictly follow the rfc and so preserve them
- //
- // See IMAP-327 and https://mailman2.u.washington.edu/mailman/htdig/imap-protocol/2010-July/001271.html
- String value = MimeUtility.unfold(header.getValue());
-
- if ("".equals(value.trim())) {
- return null;
} else {
- AddressList addressList = LenientAddressParser.DEFAULT.parseAddressList(value);
- final int size = addressList.size();
- final List<FetchResponse.Envelope.Address> addresses = new ArrayList<>(size);
- for (Address address : addressList) {
- if (address instanceof Group) {
- final Group group = (Group) address;
- addAddresses(group, addresses);
-
- } else if (address instanceof Mailbox) {
- final Mailbox mailbox = (Mailbox) address;
- final FetchResponse.Envelope.Address mailboxAddress = buildMailboxAddress(mailbox);
- addresses.add(mailboxAddress);
-
- } else {
- LOGGER.warn("Unknown address type {}", address.getClass());
- }
- }
-
- return addresses.toArray(FetchResponse.Envelope.Address[]::new);
+ LOGGER.warn("Unknown address type {}", address.getClass());
}
}
+ return addresses.toArray(FetchResponse.Envelope.Address[]::new);
+
}
private FetchResponse.Envelope.Address buildMailboxAddress(org.apache.james.mime4j.dom.address.Mailbox mailbox) {
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
index 0156e7f..41a1d68 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
@@ -280,12 +280,10 @@ public final class FetchResponseBuilder {
private FetchResponse.BodyElement wrapIfPartialFetch(Long firstOctet, Long numberOfOctets, FetchResponse.BodyElement fullResult) {
if (firstOctet == null) {
return fullResult;
- } else {
- final long numberOfOctetsAsLong = Objects.requireNonNullElse(numberOfOctets, Long.MAX_VALUE);
- final long firstOctetAsLong = firstOctet;
-
- return new PartialFetchBodyElement(fullResult, firstOctetAsLong, numberOfOctetsAsLong);
}
+ final long numberOfOctetsAsLong = Objects.requireNonNullElse(numberOfOctets, Long.MAX_VALUE);
+ final long firstOctetAsLong = firstOctet;
+ return new PartialFetchBodyElement(fullResult, firstOctetAsLong, numberOfOctetsAsLong);
}
private FetchResponse.BodyElement text(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
@@ -301,9 +299,8 @@ public final class FetchResponseBuilder {
} catch (IOException e) {
throw new MailboxException("Unable to get TEXT of body", e);
}
- } else {
- return messageResult.getBody(path.get());
}
+ return messageResult.getBody(path.get());
}
private FetchResponse.BodyElement mimeHeaders(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
@@ -349,18 +346,15 @@ public final class FetchResponseBuilder {
if (messageResult.getSize() - element.size() <= 0) {
// Seems like this mail has no body
element.noBody();
-
}
} catch (IOException e) {
throw new MailboxException("Unable to get size of header body element", e);
-
}
return element;
- } else {
- final Iterator<Header> headers = getHeaders(messageResult, path);
- List<Header> lines = MessageResultUtils.getAll(headers);
- return headerBodyElement(messageResult, name, lines, path);
}
+ final Iterator<Header> headers = getHeaders(messageResult, path);
+ List<Header> lines = MessageResultUtils.getAll(headers);
+ return headerBodyElement(messageResult, name, lines, path);
}
private FetchResponse.BodyElement fieldsNot(MessageResult messageResult, String name, Optional<MimePath> path, Collection<String> names) throws MailboxException {
@@ -379,9 +373,8 @@ public final class FetchResponseBuilder {
private Iterator<Header> getHeaders(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
if (path.isEmpty()) {
return messageResult.getHeaders().headers();
- } else {
- return messageResult.iterateHeaders(path.get());
}
+ return messageResult.iterateHeaders(path.get());
}
private Iterator<Header> getMimeHeaders(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
@@ -402,8 +395,7 @@ public final class FetchResponseBuilder {
} catch (IOException e) {
throw new MailboxException("Unable to get content", e);
}
- } else {
- return messageResult.getMimeBody(path.get());
}
+ return messageResult.getMimeBody(path.get());
}
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 14/31: [REFACTORING] Avoid variable reallocation in
PartialFetchBodyElement
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit a2ba8b1100485f63b5977aa78bb22dfdfd7e4940
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:34:54 2020 +0700
[REFACTORING] Avoid variable reallocation in PartialFetchBodyElement
---
.../james/imap/processor/fetch/PartialFetchBodyElement.java | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
index 0915a8d..78b3f8d 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/PartialFetchBodyElement.java
@@ -55,15 +55,13 @@ final class PartialFetchBodyElement implements BodyElement {
public long size() throws IOException {
final long size = delegate.size();
final long lastOctet = this.numberOfOctets + firstOctet;
- final long result;
if (firstOctet > size) {
- result = 0;
+ return 0;
} else if (size > lastOctet) {
- result = numberOfOctets;
+ return numberOfOctets;
} else {
- result = size - firstOctet;
+ return size - firstOctet;
}
- return result;
}
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 06/31: JAMES-2904 Remove unused
MessageResult::hasAttachment
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit d3cf49f82f9e2feb96db86e8f4a621f41b9b356c
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Jul 15 13:22:04 2020 +0700
JAMES-2904 Remove unused MessageResult::hasAttachment
---
.../apache/james/mailbox/model/MessageResult.java | 5 ----
.../apache/james/mailbox/MailboxManagerTest.java | 31 ---------------------
.../cassandra/mail/CassandraMessageDAO.java | 6 ----
.../cassandra/mail/MessageRepresentation.java | 5 +---
.../model/openjpa/AbstractJPAMailboxMessage.java | 6 ----
.../mailbox/maildir/mail/model/MaildirMessage.java | 5 ----
.../james/mailbox/store/MessageResultImpl.java | 5 ----
.../mailbox/store/StoreMessageResultIterator.java | 5 ----
.../store/mail/model/DelegatingMailboxMessage.java | 5 ----
.../james/mailbox/store/mail/model/Message.java | 2 --
.../mail/model/impl/SimpleMailboxMessage.java | 29 ++------------------
.../store/mail/model/impl/SimpleMessage.java | 9 +-----
.../store/AbstractMessageIdManagerStorageTest.java | 32 ----------------------
13 files changed, 5 insertions(+), 140 deletions(-)
diff --git a/mailbox/api/src/main/java/org/apache/james/mailbox/model/MessageResult.java b/mailbox/api/src/main/java/org/apache/james/mailbox/model/MessageResult.java
index 0dabf05..515507b 100644
--- a/mailbox/api/src/main/java/org/apache/james/mailbox/model/MessageResult.java
+++ b/mailbox/api/src/main/java/org/apache/james/mailbox/model/MessageResult.java
@@ -163,9 +163,4 @@ public interface MessageResult extends Comparable<MessageResult> {
*/
List<MessageAttachmentMetadata> getLoadedAttachments() throws MailboxException;
- /**
- * Indicates if the message have attachments, regardless of loaded attachments.
- */
- boolean hasAttachments() throws MailboxException;
-
}
diff --git a/mailbox/api/src/test/java/org/apache/james/mailbox/MailboxManagerTest.java b/mailbox/api/src/test/java/org/apache/james/mailbox/MailboxManagerTest.java
index c1692e9..78d528e 100644
--- a/mailbox/api/src/test/java/org/apache/james/mailbox/MailboxManagerTest.java
+++ b/mailbox/api/src/test/java/org/apache/james/mailbox/MailboxManagerTest.java
@@ -64,7 +64,6 @@ import org.apache.james.mailbox.exception.TooLongMailboxNameException;
import org.apache.james.mailbox.extension.PreDeletionHook;
import org.apache.james.mailbox.mock.DataProvisioner;
import org.apache.james.mailbox.model.ComposedMessageId;
-import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.MailboxACL;
import org.apache.james.mailbox.model.MailboxAnnotation;
@@ -76,7 +75,6 @@ import org.apache.james.mailbox.model.MailboxPath;
import org.apache.james.mailbox.model.MessageId;
import org.apache.james.mailbox.model.MessageRange;
import org.apache.james.mailbox.model.MessageResult;
-import org.apache.james.mailbox.model.MessageResultIterator;
import org.apache.james.mailbox.model.MultimailboxesSearchQuery;
import org.apache.james.mailbox.model.Quota;
import org.apache.james.mailbox.model.QuotaRoot;
@@ -93,7 +91,6 @@ import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
-import com.github.fge.lambdas.Throwing;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
@@ -2721,34 +2718,6 @@ public abstract class MailboxManagerTest<T extends MailboxManager> {
.collectList().block())
.isEmpty();
}
-
- @Test
- void getMessagesShouldIncludeHasAttachmentInformation() throws Exception {
- ComposedMessageId composeId = inboxManager.appendMessage(AppendCommand.builder()
- .withFlags(new Flags(Flags.Flag.DELETED))
- .build(ClassLoaderUtils.getSystemResourceAsSharedStream("eml/twoAttachmentsApi.eml")), session).getId();
-
- MessageResultIterator messages = inboxManager.getMessages(MessageRange.one(composeId.getUid()), FetchGroup.MINIMAL, session);
-
- assertThat(messages).toIterable()
- .hasSize(1)
- .first()
- .satisfies(Throwing.consumer(messageResult -> assertThat(messageResult.hasAttachments()).isTrue()));
- }
-
- @Test
- void getMessagesShouldNotIncludeAttachmentInformationWhenNone() throws Exception {
- ComposedMessageId composeId = inboxManager.appendMessage(AppendCommand.builder()
- .withFlags(new Flags(Flags.Flag.DELETED))
- .build(message), session).getId();
-
- MessageResultIterator messages = inboxManager.getMessages(MessageRange.one(composeId.getUid()), FetchGroup.MINIMAL, session);
-
- assertThat(messages).toIterable()
- .hasSize(1)
- .first()
- .satisfies(Throwing.consumer(messageResult -> assertThat(messageResult.hasAttachments()).isFalse()));
- }
}
@Nested
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageDAO.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageDAO.java
index 8c447d7..01ca6b9 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageDAO.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageDAO.java
@@ -263,7 +263,6 @@ public class CassandraMessageDAO {
row.getInt(BODY_START_OCTET),
new SharedByteArrayInputStream(content),
getPropertyBuilder(row),
- hasAttachment(row),
getAttachments(row).collect(Guavate.toImmutableList())));
}
@@ -285,11 +284,6 @@ public class CassandraMessageDAO {
return attachmentByIds(udtValues);
}
- private boolean hasAttachment(Row row) {
- List<UDTValue> udtValues = row.getList(ATTACHMENTS, UDTValue.class);
- return !udtValues.isEmpty();
- }
-
private Stream<MessageAttachmentRepresentation> attachmentByIds(List<UDTValue> udtValues) {
return udtValues.stream()
.map(this::messageAttachmentByIdFrom);
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/MessageRepresentation.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/MessageRepresentation.java
index d627f2b..7a9a0b1 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/MessageRepresentation.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/MessageRepresentation.java
@@ -37,18 +37,16 @@ public class MessageRepresentation {
private final Integer bodySize;
private final SharedByteArrayInputStream content;
private final PropertyBuilder propertyBuilder;
- private final boolean hasAttachment;
private final List<MessageAttachmentRepresentation> attachments;
public MessageRepresentation(MessageId messageId, Date internalDate, Long size, Integer bodySize, SharedByteArrayInputStream content,
- PropertyBuilder propertyBuilder, boolean hasAttachment, List<MessageAttachmentRepresentation> attachments) {
+ PropertyBuilder propertyBuilder, List<MessageAttachmentRepresentation> attachments) {
this.messageId = messageId;
this.internalDate = internalDate;
this.size = size;
this.bodySize = bodySize;
this.content = content;
this.propertyBuilder = propertyBuilder;
- this.hasAttachment = hasAttachment;
this.attachments = attachments;
}
@@ -65,7 +63,6 @@ public class MessageRepresentation {
.flags(metadata.getFlags())
.propertyBuilder(propertyBuilder)
.addAttachments(attachments)
- .hasAttachment(hasAttachment)
.build();
}
diff --git a/mailbox/jpa/src/main/java/org/apache/james/mailbox/jpa/mail/model/openjpa/AbstractJPAMailboxMessage.java b/mailbox/jpa/src/main/java/org/apache/james/mailbox/jpa/mail/model/openjpa/AbstractJPAMailboxMessage.java
index 004c317..480989a 100644
--- a/mailbox/jpa/src/main/java/org/apache/james/mailbox/jpa/mail/model/openjpa/AbstractJPAMailboxMessage.java
+++ b/mailbox/jpa/src/main/java/org/apache/james/mailbox/jpa/mail/model/openjpa/AbstractJPAMailboxMessage.java
@@ -523,10 +523,4 @@ public abstract class AbstractJPAMailboxMessage implements MailboxMessage {
private AttachmentId generateFixedAttachmentId(int position) {
return AttachmentId.from(getMailboxId().serialize() + "-" + getUid().asLong() + "-" + position);
}
-
- @Override
- public boolean hasAttachment() {
- return !getAttachments().isEmpty();
- }
-
}
diff --git a/mailbox/maildir/src/main/java/org/apache/james/mailbox/maildir/mail/model/MaildirMessage.java b/mailbox/maildir/src/main/java/org/apache/james/mailbox/maildir/mail/model/MaildirMessage.java
index d094699..3861dd9 100644
--- a/mailbox/maildir/src/main/java/org/apache/james/mailbox/maildir/mail/model/MaildirMessage.java
+++ b/mailbox/maildir/src/main/java/org/apache/james/mailbox/maildir/mail/model/MaildirMessage.java
@@ -292,9 +292,4 @@ public class MaildirMessage implements Message {
return AttachmentId.from(messageName.getFullName() + "-" + position);
}
- @Override
- public boolean hasAttachment() {
- return !getAttachments().isEmpty();
- }
-
}
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/MessageResultImpl.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/MessageResultImpl.java
index 6d68e3d..b52a655 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/MessageResultImpl.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/MessageResultImpl.java
@@ -310,11 +310,6 @@ public class MessageResultImpl implements MessageResult {
return message.getAttachments();
}
- @Override
- public boolean hasAttachments() {
- return message.hasAttachment();
- }
-
private static final class HeadersImpl implements Headers {
private final MailboxMessage msg;
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageResultIterator.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageResultIterator.java
index 02a3b19..2fcd537 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageResultIterator.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageResultIterator.java
@@ -227,11 +227,6 @@ public class StoreMessageResultIterator implements MessageResultIterator {
}
@Override
- public boolean hasAttachments() throws MailboxException {
- throw exception;
- }
-
- @Override
public int compareTo(MessageResult that) {
return getUid().compareTo(that.getUid());
}
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/DelegatingMailboxMessage.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/DelegatingMailboxMessage.java
index 398ec51..cdd3ff2 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/DelegatingMailboxMessage.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/DelegatingMailboxMessage.java
@@ -119,9 +119,4 @@ public abstract class DelegatingMailboxMessage implements MailboxMessage {
public List<MessageAttachmentMetadata> getAttachments() {
return message.getAttachments();
}
-
- @Override
- public boolean hasAttachment() {
- return message.hasAttachment();
- }
}
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/Message.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/Message.java
index c2e3fb8..9cacbfc 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/Message.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/Message.java
@@ -109,6 +109,4 @@ public interface Message {
*/
List<MessageAttachmentMetadata> getAttachments();
- boolean hasAttachment();
-
}
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
index c19d5c2..32be330 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMailboxMessage.java
@@ -65,7 +65,6 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
private Optional<MessageUid> uid = Optional.empty();
private Optional<ModSeq> modseq = Optional.empty();
private ImmutableList.Builder<MessageAttachmentMetadata> attachments = ImmutableList.builder();
- private Optional<Boolean> hasAttachment = Optional.empty();
public Builder messageId(MessageId messageId) {
this.messageId = messageId;
@@ -104,16 +103,6 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
return this;
}
- public Builder hasAttachment() {
- this.hasAttachment = Optional.of(true);
- return this;
- }
-
- public Builder hasAttachment(boolean hasAttachment) {
- this.hasAttachment = Optional.of(hasAttachment);
- return this;
- }
-
public Builder flags(Flags flags) {
this.flags = flags;
return this;
@@ -145,9 +134,8 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
Preconditions.checkNotNull(mailboxId, "mailboxId is required");
ImmutableList<MessageAttachmentMetadata> attachments = this.attachments.build();
- boolean hasAttachment = this.hasAttachment.orElse(!attachments.isEmpty());
SimpleMailboxMessage simpleMailboxMessage = new SimpleMailboxMessage(messageId, internalDate, size,
- bodyStartOctet, content, flags, propertyBuilder, mailboxId, attachments, hasAttachment);
+ bodyStartOctet, content, flags, propertyBuilder, mailboxId, attachments);
uid.ifPresent(simpleMailboxMessage::setUid);
modseq.ifPresent(simpleMailboxMessage::setModSeq);
@@ -175,7 +163,6 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
.internalDate(original.getInternalDate())
.size(original.getFullContentOctets())
.flags(original.createFlags())
- .hasAttachment(original.hasAttachment())
.propertyBuilder(propertyBuilder);
}
@@ -205,8 +192,7 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
public SimpleMailboxMessage(MessageId messageId, Date internalDate, long size, int bodyStartOctet,
SharedInputStream content, Flags flags,
- PropertyBuilder propertyBuilder, MailboxId mailboxId, List<MessageAttachmentMetadata> attachments,
- boolean hasAttachment) {
+ PropertyBuilder propertyBuilder, MailboxId mailboxId, List<MessageAttachmentMetadata> attachments) {
super(new SimpleMessage(
messageId,
content, size, internalDate, propertyBuilder.getSubType(),
@@ -214,8 +200,7 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
bodyStartOctet,
propertyBuilder.getTextualLineCount(),
propertyBuilder.toProperties(),
- attachments,
- hasAttachment));
+ attachments));
setFlags(flags);
this.mailboxId = mailboxId;
@@ -223,14 +208,6 @@ public class SimpleMailboxMessage extends DelegatingMailboxMessage {
}
public SimpleMailboxMessage(MessageId messageId, Date internalDate, long size, int bodyStartOctet,
- SharedInputStream content, Flags flags,
- PropertyBuilder propertyBuilder, MailboxId mailboxId, List<MessageAttachmentMetadata> attachments) {
- this(messageId, internalDate, size, bodyStartOctet,
- content, flags,
- propertyBuilder, mailboxId, attachments, !attachments.isEmpty());
- }
-
- public SimpleMailboxMessage(MessageId messageId, Date internalDate, long size, int bodyStartOctet,
SharedInputStream content, Flags flags,
PropertyBuilder propertyBuilder, MailboxId mailboxId) {
this(messageId, internalDate, size, bodyStartOctet,
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
index 36042a3..62a5b86 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
@@ -42,9 +42,8 @@ public class SimpleMessage implements Message {
private final Long textualLineCount;
private final List<Property> properties;
private final List<MessageAttachmentMetadata> attachments;
- private final boolean hasAttachments;
- public SimpleMessage(MessageId messageId, SharedInputStream content, long size, Date internalDate, String subType, String mediaType, int bodyStartOctet, Long textualLineCount, List<Property> properties, List<MessageAttachmentMetadata> attachments, boolean hasAttachments) {
+ public SimpleMessage(MessageId messageId, SharedInputStream content, long size, Date internalDate, String subType, String mediaType, int bodyStartOctet, Long textualLineCount, List<Property> properties, List<MessageAttachmentMetadata> attachments) {
this.messageId = messageId;
this.subType = subType;
this.mediaType = mediaType;
@@ -55,7 +54,6 @@ public class SimpleMessage implements Message {
this.textualLineCount = textualLineCount;
this.properties = properties;
this.attachments = attachments;
- this.hasAttachments = hasAttachments;
}
@Override
@@ -126,9 +124,4 @@ public class SimpleMessage implements Message {
public List<MessageAttachmentMetadata> getAttachments() {
return attachments;
}
-
- @Override
- public boolean hasAttachment() {
- return hasAttachments;
- }
}
diff --git a/mailbox/store/src/test/java/org/apache/james/mailbox/store/AbstractMessageIdManagerStorageTest.java b/mailbox/store/src/test/java/org/apache/james/mailbox/store/AbstractMessageIdManagerStorageTest.java
index 4c7f73a..8416d42 100644
--- a/mailbox/store/src/test/java/org/apache/james/mailbox/store/AbstractMessageIdManagerStorageTest.java
+++ b/mailbox/store/src/test/java/org/apache/james/mailbox/store/AbstractMessageIdManagerStorageTest.java
@@ -49,11 +49,9 @@ import org.apache.james.mailbox.model.MailboxACL.Right;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
import org.apache.james.mailbox.model.MessageResult;
-import org.apache.james.util.ClassLoaderUtils;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
-import com.github.fge.lambdas.Throwing;
import com.github.steveash.guavate.Guavate;
import com.google.common.collect.ImmutableList;
@@ -981,34 +979,4 @@ public abstract class AbstractMessageIdManagerStorageTest {
.extracting(MessageResult::getFlags)
.containsOnly(flags);
}
-
- @Test
- void getMessagesShouldIncludeAttachmentInformation() throws Exception {
- MessageId messageId = testingData.getMailboxManager().getMailbox(bobMailbox1.getMailboxId(), bobSession)
- .appendMessage(MessageManager.AppendCommand.builder()
- .withFlags(new Flags(Flags.Flag.DELETED))
- .build(ClassLoaderUtils.getSystemResourceAsSharedStream("eml/twoAttachmentsApi.eml")), bobSession)
- .getId()
- .getMessageId();
-
- List<MessageResult> messages = messageIdManager.getMessage(messageId, FetchGroup.MINIMAL, bobSession);
-
- assertThat(messages)
- .hasSize(1)
- .first()
- .satisfies(Throwing.consumer(messageResult -> assertThat(messageResult.hasAttachments()).isTrue()));
- }
-
- @Test
- void getMessagesShouldNotIncludeAttachmentInformationWhenNone() throws Exception {
- Flags flags = new Flags(Flags.Flag.FLAGGED);
- MessageId messageId = testingData.persist(bobMailbox1.getMailboxId(), messageUid1, flags, bobSession);
-
- List<MessageResult> messages = messageIdManager.getMessage(messageId, FetchGroup.MINIMAL, bobSession);
-
- assertThat(messages)
- .hasSize(1)
- .first()
- .satisfies(Throwing.consumer(messageResult -> assertThat(messageResult.hasAttachments()).isFalse()));
- }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 16/31: [REFACTORING] MimeBodyElement should use
StandardCharset
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 0779deadcfc42a78ffe4c3d91dfcf99af372b8c0
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:40:58 2020 +0700
[REFACTORING] MimeBodyElement should use StandardCharset
---
.../java/org/apache/james/imap/processor/fetch/MimeBodyElement.java | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
index 9991dcf..715674c 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/MimeBodyElement.java
@@ -18,11 +18,12 @@
****************************************************************/
package org.apache.james.imap.processor.fetch;
+import static java.nio.charset.StandardCharsets.US_ASCII;
+
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
-import java.nio.charset.Charset;
import java.util.List;
import org.apache.james.imap.api.ImapConstants;
@@ -30,7 +31,6 @@ import org.apache.james.imap.message.response.FetchResponse.BodyElement;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.model.Header;
-
/**
* {@link BodyElement} which represent a MIME element specified by for example (BODY[1.MIME])
*
@@ -41,8 +41,6 @@ public class MimeBodyElement implements BodyElement {
protected final List<Header> headers;
protected long size;
- private static final Charset US_ASCII = Charset.forName("US-ASCII");
-
public MimeBodyElement(String name, List<Header> headers) throws MailboxException {
super();
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 25/31: JAMES-3302 Document logging for the
Distributed Server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e8937d93a002f3e21ca07e58a616f4734e338999
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 13:38:11 2020 +0700
JAMES-3302 Document logging for the Distributed Server
---
docs/modules/servers/nav.adoc | 1 +
.../servers/pages/distributed/operate/index.adoc | 10 +-
.../servers/pages/distributed/operate/logging.adoc | 108 +++++++++++++++++++++
3 files changed, 115 insertions(+), 4 deletions(-)
diff --git a/docs/modules/servers/nav.adoc b/docs/modules/servers/nav.adoc
index ffd4c6e..78e7f7b 100644
--- a/docs/modules/servers/nav.adoc
+++ b/docs/modules/servers/nav.adoc
@@ -11,6 +11,7 @@
*** xref:distributed/configure/index.adoc[]
*** xref:distributed/operate/index.adoc[]
**** xref:distributed/operate/guide.adoc[]
+**** xref:distributed/operate/logging.adoc[]
**** xref:distributed/operate/webadmin.adoc[]
**** xref:distributed/operate/metrics.adoc[]
**** xref:distributed/operate/cli.adoc[]
diff --git a/docs/modules/servers/pages/distributed/operate/index.adoc b/docs/modules/servers/pages/distributed/operate/index.adoc
index 4c62a69..8b20adb 100644
--- a/docs/modules/servers/pages/distributed/operate/index.adoc
+++ b/docs/modules/servers/pages/distributed/operate/index.adoc
@@ -6,15 +6,17 @@ Once you have a Distributed James server up and running you then need to ensure
You may also need to perform some operation maintenance or recover from incidents. This section covers
these topics.
-The xref:main:servers:distributed:operate:webadmin.adoc[WebAdmin Restfull administration API] is the
+Read more about link:logging.adoc[Logging].
+
+The link:webadmin.adoc[WebAdmin Restfull administration API] is the
recommended way to operate the Distributed James server. It allows managing and interacting with most
server components.
-The xref:main:servers:distributed:operate:cli.adoc[Command line interface] allows to interact with some
+The link:cli.adoc[Command line interface] allows to interact with some
server components. However it relies on JMX technologies and its use is discouraged.
-The xref:main:servers:distributed:operate:metrics.adoc[metrics] allows to build latency and throughput
+The link:metrics.adoc[metrics] allows to build latency and throughput
graphs, that can be visualized, for instance in *Grafana*.
-Finally, we did put together a xref:main:servers:distributed:operate:metrics.adoc[detailed guide] for
+Finally, we did put together a link:metrics.adoc[detailed guide] for
distributed James operators.
diff --git a/docs/modules/servers/pages/distributed/operate/logging.adoc b/docs/modules/servers/pages/distributed/operate/logging.adoc
new file mode 100644
index 0000000..e092b8f
--- /dev/null
+++ b/docs/modules/servers/pages/distributed/operate/logging.adoc
@@ -0,0 +1,108 @@
+= Logging
+
+We recommend to closely monitoring *ERROR* and *WARNING* logs. Those
+logs should be considered not normal.
+
+If you encounter some suspicious logs:
+
+* If you have any doubt about the log being caused by a bug in James
+source code, please reach us via the bug tracker, the user mailing list or our Gitter channel (see our
+http://james.apache.org/#second[community page])
+* They can be due to insufficient performance from tier applications (eg
+Cassandra timeouts). In such case we advise you to conduct a close
+review of performances at the tier level.
+
+Leveraging filters in Kibana discover view can help filtering out
+``already known'' frequently occurring logs.
+
+When reporting ERROR or WARNING logs, consider adding the full logs, and
+related data (eg the raw content of a mail triggering an issue) to the
+bug report in order to ease resolution.
+
+== Logging configuration
+
+Distributed James uses link:http://logback.qos.ch/[logback] as a logging library.
+
+Information about logback configuration can be found
+link:http://logback.qos.ch/manual/configuration.html[here].
+
+== Structured logging
+
+Distributed Server leverage the use of MDC in order to achieve structured logging,
+and better add context to the logged information. We furthermore ship
+link:https://github.com/linagora/logback-elasticsearch-appender[Logback Elasticsearch Appender]
+on the classpath to easily allow direct log indexation in
+link:https://www.elastic.co/elasticsearch[ElasticSearch].
+
+Here is a sample `conf/logback.xml` configuration file for logback with the following
+pre-requisites:
+
+* Logging both in an unstructured fashion on the console and in a structure fashion in
+ElasticSearch
+* Logging ElasticSearch Log appender logs in the console
+
+....
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration scan="true" scanPeriod="30 seconds">
+
+ <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
+ <resetJUL>true</resetJUL>
+ </contextListener>
+
+ <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
+ <encoder>
+ <pattern>%d{yyyy.MM.dd HH:mm:ss.SSS} %highlight([%-5level]) %logger{15} - %msg%n%rEx</pattern>
+ <immediateFlush>false</immediateFlush>
+ </encoder>
+ </appender>
+
+ <appender name="ELASTIC" class="com.linagora.logback.elasticsearch.ElasticsearchAppender">
+ <url>http://elasticsearch:9200/_bulk</url>
+ <index>logs-james-%date{yyyy.MM.dd}</index>
+ <type>tester</type>
+ <includeMdc>true</includeMdc>
+ <excludedMdcKeys>host</excludedMdcKeys>
+ <errorLoggerName>es-error-logger</errorLoggerName>
+ <properties>
+ <property>
+ <name>host</name>
+ <value>${HOSTNAME}</value>
+ <allowEmpty>false</allowEmpty>
+ </property>
+ <property>
+ <name>severity</name>
+ <value>%level</value>
+ </property>
+ <property>
+ <name>thread</name>
+ <value>%thread</value>
+ </property>
+ <property>
+ <name>stacktrace</name>
+ <value>%ex</value>
+ </property>
+ <property>
+ <name>logger</name>
+ <value>%logger</value>
+ </property>
+ </properties>
+ <headers>
+ <header>
+ <name>Content-Type</name>
+ <value>application/json</value>
+ </header>
+ </headers>
+ </appender>
+
+ <root level="WARN">
+ <appender-ref ref="ELASTIC" />
+ </root>
+
+ <logger name="es-error-logger" level="DEBUG" additivity="false">
+ <appender-ref ref="CONSOLE" />
+ </logger>
+
+ <logger name="org.apache.james" level="INFO" />
+
+</configuration>
+....
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 28/31: JAMES-3302 Write more about Architecture for
Distributed server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 9d217f1998d47f9631e12856719d70e2962a5710
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 17:31:47 2020 +0700
JAMES-3302 Write more about Architecture for Distributed server
- Mention protocols
- List (and detail some) components
---
.../servers/pages/distributed/architecture.adoc | 81 +++++++++++++++++++++-
1 file changed, 80 insertions(+), 1 deletion(-)
diff --git a/docs/modules/servers/pages/distributed/architecture.adoc b/docs/modules/servers/pages/distributed/architecture.adoc
index d4e1678..cb4b33c 100644
--- a/docs/modules/servers/pages/distributed/architecture.adoc
+++ b/docs/modules/servers/pages/distributed/architecture.adoc
@@ -17,6 +17,20 @@ In order to deliver its promises, the Distributed Server leverages the following
* *Tika* (optional) enables text extraction from attachments, thus improving full text search results.
* *SpamAssassin* (optional) can be used for Spam detection and user feedback is supported.
+== Protocols
+
+The following protocols are supported and can be used to interact with the Distributed Server:
+
+* *SMTP*
+* *IMAP*
+* link:operate/webadmin.adoc[WebAdmin] REST Administration API
+* *LMTP*
+
+The following protocols should be considered experimental
+
+* *JMAP* (draft specification as defined link:https://github.com/apache/james-project/tree/master/server/protocols/jmap-draft/doc[here])
+* *POP3*
+
== Components
This section presents the various components of the Distributed server, providing context about
@@ -108,7 +122,25 @@ delivery retries, or
=== Mailbox
-(TODO)
+Storage for emails belonging for users.
+
+Metadata are stored in Cassandra while headers, bodies and attachments are stored
+within the xref:#_blobstore[BlobStore].
+
+==== Search index
+
+Emails are indexed asynchronously in ElasticSearch via the xref:#_event_bus[EventBus]
+in order to enpower advanced and fast email full text search.
+
+Text extraction can be set up using link:https://tika.apache.org/[Tika], allowing
+to extract the text from attachment, allowing to search your emails based on the attachment
+textual content. In such case, the ElasticSearch indexer will call a Tika server prior
+indexing.
+
+==== Quotas
+
+Current Quotas of users are hold in a Cassandra projection. Limitations can be defined via
+user, domain or globally.
==== Event Bus
@@ -163,3 +195,50 @@ However, mails are not retained forever as you have to configure a
retention period before using it (with one-year retention by default if
not defined). It’s also possible to permanently delete a mail if needed.
+=== Data
+
+(TODO)
+
+=== Recipient rewrite tables
+
+(TODO)
+
+=== BlobStore
+
+Stores potentially large binary data.
+
+Mailbox component, Mail Queue component, Deleted Message Vault
+component relies on it.
+
+Supported backends includes ObjectStorage (link:https://wiki.openstack.org/wiki/Swift[Swift], S3 API).
+
+Encryption can be configured on top of ObjectStorage.
+
+Blobs are currently deduplicated in order to reduce storage space. This means that two blobs with
+the same content will be stored one once.
+
+The downside is that deletion is more complicated, and a garbage collection needs to be run. This is a work
+in progress. See link:https://issues.apache.org/jira/browse/JAMES-3150[JAMES-3150].
+
+=== Task Manager
+
+Allows to control and schedule long running tasks run by other
+components. Among other it enables scheduling, progress monitoring,
+cancelation of long running tasks.
+
+Distributed James leverage a task manager using Event Sourcing and RabbitMQ for messaging.
+
+=== Event sourcing
+
+link:https://martinfowler.com/eaaDev/EventSourcing.html[Event sourcing] implementation
+for the Distributed server stores events in Cassandra. It enables components
+to rely on event sourcing technics for taking decisions.
+
+A short list of usage are:
+
+* Data leak prevention storage
+* JMAP filtering rules storage
+* Validation of the MailQueue configuration
+* Sending email warnings to user close to their quota
+* Implementation of the TaskManager
+
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 30/31: JAMES-3302 Fixing dead links for Distributed
Server documentation
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1c9f1d3080b896a787665e1c7f8f9d9b5e21a022
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 17:52:41 2020 +0700
JAMES-3302 Fixing dead links for Distributed Server documentation
---
docs/modules/servers/pages/distributed.adoc | 12 +--
.../servers/pages/distributed/architecture.adoc | 2 +-
.../servers/pages/distributed/extend/index.adoc | 8 +-
.../servers/pages/distributed/operate/guide.adoc | 91 +++++++++++-----------
.../servers/pages/distributed/operate/index.adoc | 10 +--
5 files changed, 60 insertions(+), 63 deletions(-)
diff --git a/docs/modules/servers/pages/distributed.adoc b/docs/modules/servers/pages/distributed.adoc
index 9f298bf..5ce28f2 100644
--- a/docs/modules/servers/pages/distributed.adoc
+++ b/docs/modules/servers/pages/distributed.adoc
@@ -13,9 +13,9 @@ This server is:
* The most feature-rich server, but also by far the most complex
You will find information about its
-xref:main:servers:distributed:architecture.adoc[architecture], how to
-xref:main:servers:distributed:run.adoc[run it], how to
-xref:main:servers:distributed:run-docker.adoc[run it with Docker], how to
-xref:main:servers:distributed:configure/index.adoc[configure it], how to
-xref:main:servers:distributed:operate/index.adoc[operate it], how to
-xref:main:servers:distributed:extend/index.adoc[extend it].
+xref:distributed/architecture.adoc[architecture], how to
+xref:distributed/run.adoc[run it], how to
+xref:distributed/run-docker.adoc[run it with Docker], how to
+xref:distributed/configure/index.adoc[configure it], how to
+xref:distributed/operate/index.adoc[operate it], how to
+xref:distributed/extend/index.adoc[extend it].
diff --git a/docs/modules/servers/pages/distributed/architecture.adoc b/docs/modules/servers/pages/distributed/architecture.adoc
index cb4b33c..bf08326 100644
--- a/docs/modules/servers/pages/distributed/architecture.adoc
+++ b/docs/modules/servers/pages/distributed/architecture.adoc
@@ -23,7 +23,7 @@ The following protocols are supported and can be used to interact with the Distr
* *SMTP*
* *IMAP*
-* link:operate/webadmin.adoc[WebAdmin] REST Administration API
+* xref:distributed/operate/webadmin.adoc[WebAdmin] REST Administration API
* *LMTP*
The following protocols should be considered experimental
diff --git a/docs/modules/servers/pages/distributed/extend/index.adoc b/docs/modules/servers/pages/distributed/extend/index.adoc
index ee0b7e2..d633912 100644
--- a/docs/modules/servers/pages/distributed/extend/index.adoc
+++ b/docs/modules/servers/pages/distributed/extend/index.adoc
@@ -4,15 +4,15 @@ The Distributed Server exposes several interfaces allowing the user to write cus
order to extend the Distributed Server behavior.
Writing *Mailets* and *Matchers* allows one to supply custom components for the
-xref:main:servers:distributed:extend:mail-processing.adoc[Mail Processing] and
+xref:distributed/extend/mail-processing.adoc[Mail Processing] and
enables to take decisions, and implement your business logic at the transport level.
-Writing xref:main:servers:distributed:extend:mailbox-listeners.adoc[Mailbox listeners] enables to
+Writing xref:distributed/extend/mailbox-listeners.adoc[Mailbox listeners] enables to
react to your user interaction with their mailbox. This powerful tool allows build advanced features
for mail delivery servers.
-Writing xref:main:servers:distributed:extend:smtp-hooks.adoc[SMTP hookd] enables to
+Writing xref:distributed/extend/smtp-hooks.adoc[SMTP hookd] enables to
add features to your SMTP server.
-Writing xref:main:servers:distributed:extend:webadmin-routes.adoc[WebAdmin routes] enables to
+Writing xref:distributed/extend/webadmin-routes.adoc[WebAdmin routes] enables to
add features to the WebAdmin REST API.
diff --git a/docs/modules/servers/pages/distributed/operate/guide.adoc b/docs/modules/servers/pages/distributed/operate/guide.adoc
index 6f6f921..b9d4868 100644
--- a/docs/modules/servers/pages/distributed/operate/guide.adoc
+++ b/docs/modules/servers/pages/distributed/operate/guide.adoc
@@ -19,9 +19,9 @@ considered experimental and thus targets advanced users.
A toolbox is available to help an administrator diagnose issues:
-* xref:#logging.adoc[Structured logging into Kibana]
-* link:#metrics.adoc[Metrics graphs into Grafana]
-* xref:manage-webadmin.adoc#_healthcheck[WebAdmin HealthChecks]
+* xref:distributed/operate/logging.adoc[Structured logging into Kibana]
+* xref:distributed/operate/metrics.adoc[Metrics graphs into Grafana]
+* xref:distributed/operate/webadmin.adoc#_healthcheck[WebAdmin HealthChecks]
== Mail processing
@@ -34,15 +34,15 @@ Furthermore, given the default mailet container configuration, we recommend moni
`cassandra://var/mail/error/` to be empty.
WebAdmin exposes all utilities for
-xref:manage-webadmin.html#_reprocessing_mails_from_a_mail_repository[reprocessing
+xref:distributed/operate/webadmin.adoc#_reprocessing_mails_from_a_mail_repository[reprocessing
all mails in a mail repository] or
-xref:manage-webadmin.html#_reprocessing_a_specific_mail_from_a_mail_repository[reprocessing
+xref:distributed/operate/webadmin.adoc#_reprocessing_a_specific_mail_from_a_mail_repository[reprocessing
a single mail in a mail repository].
Also, one can decide to
-xref:manage-webadmin.html#_removing_all_mails_from_a_mail_repository[delete
+xref:distributed/operate/webadmin.adoc#_removing_all_mails_from_a_mail_repository[delete
all the mails of a mail repository] or
-xref:manage-webadmin.html#_removing_a_mail_from_a_mail_repository[delete
+xref:distributed/operate/webadmin.adoc#_removing_a_mail_from_a_mail_repository[delete
a single mail of a mail repository].
Performance of mail processing can be monitored via the
@@ -59,17 +59,15 @@ to emails being stored in `cassandra://var/mail/rrt-error/`.
We recommend monitoring the content of this mail repository to be empty.
If it is not empty, we recommend
-verifying user mappings via xref:webadmin.adoc#_user_mappings[User
-Mappings webadmin API] then once identified break the loop by removing
+verifying user mappings via xref:distributed/operate/webadmin.adoc#_user_mappings[User Mappings webadmin API] then once identified break the loop by removing
some Recipient Rewrite Table entry via the
-xref:webadmin.adoc#_removing_an_alias_of_an_user[Delete Alias],
-xref:webadmin.adoc#_removing_a_group_member[Delete Group member],
-xref:webadmin.adoc#_removing_a_destination_of_a_forward[Delete
-forward], xref:webadmin.adoc#_remove_an_address_mapping[Delete
-Address mapping],
-xref:webadmin.adoc#_removing_a_domain_mapping[Delete Domain
-mapping] or xref:webadmin.adoc#_removing_a_regex_mapping[Delete
-Regex mapping] APIs (as needed).
+xref:distributed/operate/webadmin.adoc#_removing_an_alias_of_an_user[Delete Alias],
+xref:distributed/operate/webadmin.adoc#_removing_a_group_member[Delete Group member],
+xref:distributed/operate/webadmin.adoc#_removing_a_destination_of_a_forward[Delete forward],
+xref:distributed/operate/webadmin.adoc#_remove_an_address_mapping[Delete Address mapping],
+xref:distributed/operate/webadmin.adoc#_removing_a_domain_mapping[Delete Domain mapping]
+or xref:distributed/operate/webadmin.adoc#_removing_a_regex_mapping[Delete Regex mapping]
+APIs (as needed).
The `Mail.error` field can help diagnose the issue as well. Then once
the root cause has been addressed, the mail can be reprocessed.
@@ -95,31 +93,31 @@ grafana board].
Upon exceptions, a bounded number of retries are performed (with
exponential backoff delays). If after those retries the listener is
still failing to perform its operation, then the event will be stored in
-the xref:webadmin.adoc#_event_dead_letter[Event Dead Letter]. This
+the xref:distributed/operate/webadmin.adoc#_event_dead_letter[Event Dead Letter]. This
API allows diagnosing issues, as well as redelivering the events.
To check that you have undelivered events in your system, you can first
run the associated with
-xref:webadmin.adoc#_healthcheck[event dead letter health check] .
+xref:distributed/operate/webadmin.adoc#_healthcheck[event dead letter health check] .
You can explore Event DeadLetter content through WebAdmin. For
-this, xref:webadmin.adoc#_listing_mailbox_listener_groups[list mailbox listener groups]
+this, xref:distributed/operate/webadmin.adoc#_listing_mailbox_listener_groups[list mailbox listener groups]
you will get a list of groups back, allowing
you to check if those contain registered events in each by
-xref:webadmin.adoc#_listing_failed_events[listing their failed events].
+xref:distributed/operate/webadmin.adoc#_listing_failed_events[listing their failed events].
If you get failed events IDs back, you can as well
-xref:webadmin.adoc#_getting_event_details[check their details].
+xref:distributed/operate/webadmin.adoc#_getting_event_details[check their details].
An easy way to solve this is just to trigger then the
-xref:webadmin.adoc#_redeliver_all_events[redeliver all events]
+xref:distributed/operate/webadmin.adoc#_redeliver_all_events[redeliver all events]
task. It will start reprocessing all the failed events registered in
event dead letters.
If for some other reason you don’t need to redeliver all events, you
have more fine-grained operations allowing you to
-xref:webadmin.adoc#_redeliver_group_events[redeliver group events]
+xref:distributed/operate/webadmin.adoc#_redeliver_group_events[redeliver group events]
or even just
-xref:webadmin.adoc#_redeliver_a_single_event[redeliver a single event].
+xref:distributed/operate/webadmin.adoc#_redeliver_a_single_event[redeliver a single event].
== ElasticSearch Indexing
@@ -136,7 +134,7 @@ processing those events can fail sometimes.
Currently, an administrator can monitor indexation failures through
`ERROR` log review. You can as well
-xref:manage-webadmin.html#_listing_failed_events[list failed events] by
+xref:distributed/operate/webadmin.adoc#_listing_failed_events[list failed events] by
looking with the group called
`org.apache.james.mailbox.elasticsearch.events.ElasticSearchListeningMessageSearchIndex$ElasticSearchListeningMessageSearchIndexGroup`.
A first on-the-fly solution could be to just
@@ -147,13 +145,13 @@ Cassandra storage exceptions), then you might need to use our WebAdmin
reIndexing tasks.
From there, you have multiple choices. You can
-xref:webadmin.adoc#_reIndexing_all_mails[reIndex all mails],
-xref:webadmin.adoc#_reIndexing_a_mailbox_mails[reIndex mails from a mailbox] or even just
-xref:webadmin.adoc#_reIndexing_a_single_mail[reIndex a single mail].
+xref:distributed/operate/webadmin.adoc#_reindexing_all_mails[reIndex all mails],
+xref:distributed/operate/webadmin.adoc#_reindexing_a_mailbox_mails[reIndex mails from a mailbox] or even just
+xref:distributed/operate/webadmin.adoc#_reindexing_a_single_mail_by_messageid[reIndex a single mail].
When checking the result of a reIndexing task, you might have failed
reprocessed mails. You can still use the task ID to
-xref:webadmin.adoc#_fixing_previously_failed_reIndexing[reprocess previously failed reIndexing mails].
+xref:distributed/operate/webadmin.adoc#_fixing_previously_failed_reindexing[reprocess previously failed reIndexing mails].
=== On the fly ElasticSearch Index setting update
@@ -214,7 +212,7 @@ message reads and will temporary decrease the performance.
==== How to detect the outdated projections
You can watch the `MessageFastViewProjection` health check at
-xref:webadmin.adoc#_check_all_components[webadmin documentation].
+xref:distributed/operate/webadmin.adoc#_check_all_components[webadmin documentation].
It provides a check based on the ratio of missed projection reads.
==== How to solve
@@ -243,7 +241,7 @@ diagnostic and fixes.
==== How to solve
An admin can run offline webadmin
-xref:webadmin.adoc#_fixing_mailboxes_inconsistencies[solve Cassandra mailbox object inconsistencies task]
+xref:distributed/operate/webadmin.adoc#_fixing_mailboxes_inconsistencies[solve Cassandra mailbox object inconsistencies task]
in order to sanitize his
mailbox denormalization.
@@ -267,7 +265,7 @@ message prefix: `Invalid mailbox counters`.
==== How to solve
Execute the
-xref:webadmin.adoc#_recomputing_mailbox_counters[recompute Mailbox counters task].
+xref:distributed/operate/webadmin.adoc#_recomputing_mailbox_counters[recompute Mailbox counters task].
This task is not concurrent-safe. Concurrent
increments & decrements will be ignored during a single mailbox
processing. Re-running this task may eventually return the correct
@@ -287,7 +285,7 @@ User can see a message in JMAP but not in IMAP, or mark a message as
==== How to solve
Execute the
-xref:webadmin.adoc#_fixing_messages_inconsistencies[solve Cassandra message inconsistencies task]. This task is not
+xref:distributed/operate/webadmin.adoc#_fixing_message_inconsistencies[solve Cassandra message inconsistencies task]. This task is not
concurrent-safe. User actions concurrent to the inconsistency fixing
task could result in new inconsistencies being created. However the
source of truth `imapUidTable` will not be affected and thus re-running
@@ -307,7 +305,7 @@ Incorrect quotas could be seen in the `Mail User Agent` (IMAP or JMAP).
==== How to solve
Execute the
-xref:webadmin.adoc#_recomputing_current_quotas_for_users[recompute Quotas counters task]. This task is not concurrent-safe. Concurrent
+xref:distributed/operate/webadmin.adoc#_recomputing_current_quotas_for_users[recompute Quotas counters task]. This task is not concurrent-safe. Concurrent
operations will result in an invalid quota to be persisted. Re-running
this task may eventually return the correct result.
@@ -327,7 +325,7 @@ the mean time, the recommendation is to execute the
==== How to solve
Execute the Cassandra mapping `SolveInconsistencies` task described in
-xref:webadmin.adoc#_operations_on_mappings_sources[webadmin documentation]
+xref:distributed/operate/webadmin.adoc#_operations_on_mappings_sources[webadmin documentation]
== Setting Cassandra user permissions
@@ -496,17 +494,16 @@ performance issues. As such, we advise setting
Managing an email queue is an easy task if you follow this procedure:
-* First, xref:webadmin.adoc#_listing_mail_queues[List mail queues]
-and xref:webadmin.adoc#Getting_a_mail_queue_details[get a mail
-queue details].
+* First, xref:distributed/operate/webadmin.adoc#_listing_mail_queues[List mail queues]
+and xref:distributed/operate/webadmin.adoc#_getting_a_mail_queue_details[get a mail queue details].
* And then
-xref:webadmin.adoc#_listing_the_mails_of_a_mail_queue[List the mails of a mail queue].
+xref:distributed/operate/webadmin.adoc#_listing_the_mails_of_a_mail_queue[List the mails of a mail queue].
In case, you need to clear an email queue because there are only spam or
trash emails in the email queue you have this procedure to follow:
* All mails from the given mail queue will be deleted with
-xref:webadmin.adoc#_clearing_a_mail_queue[Clearing a mail queue].
+xref:distributed/operate/webadmin.adoc#_clearing_a_mail_queue[Clearing a mail queue].
== Updating Cassandra schema version
@@ -543,20 +540,20 @@ These schema updates can be triggered by webadmin using the Cassandra
backend. Following steps are for updating Cassandra schema version:
* At the very first step, you need to
-xref:webadmin.adoc#_retrieving_current_cassandra_schema_version[retrieve
+xref:distributed/operate/webadmin.adoc#_retrieving_current_cassandra_schema_version[retrieve
current Cassandra schema version]
* And then, you
-xref:webadmin.adoc#_retrieving_latest_available_cassandra_schema_version[retrieve
+xref:distributed/operate/webadmin.adoc#_retrieving_latest_available_cassandra_schema_version[retrieve
latest available Cassandra schema version] to make sure there is a
latest available version
* Eventually, you can update the current schema version to the one you
got with
-xref:webadmin.adoc#_upgrading_to_the_latest_version[upgrading to
+xref:distributed/operate/webadmin.adoc#_upgrading_to_the_latest_version[upgrading to
the latest version]
Otherwise, if you need to run the migrations to a specific version, you
can use
-xref:webadmin.adoc#_upgrading_to_a_specific_version[Upgrading to a
+xref:distributed/operate/webadmin.adoc#_upgrading_to_a_specific_version[Upgrading to a
specific version]
== Deleted Message Vault
@@ -594,13 +591,13 @@ by default if not defined).
=== Restore deleted messages after deletion
After users deleted their mails and emptied the trash, the admin can use
-xref:webadmin.adoc#_deleted-messages-vault[Restore Deleted Messages]
+xref:distributed/operate/webadmin.adoc#_restore_deleted_messagest[Restore Deleted Messages]
to restore all the deleted mails.
=== Cleaning expired deleted messages
You can delete all deleted messages older than the configured
`retentionPeriod` by using
-xref:webadmin.adoc#_deleted-messages-vault[Purge Deleted Messages].
+xref:distributed/operate/webadmin.adoc#_deleted_messages_vault[Purge Deleted Messages].
We recommend calling this API in CRON job on 1st day each
month.
diff --git a/docs/modules/servers/pages/distributed/operate/index.adoc b/docs/modules/servers/pages/distributed/operate/index.adoc
index 8b20adb..a2ebd06 100644
--- a/docs/modules/servers/pages/distributed/operate/index.adoc
+++ b/docs/modules/servers/pages/distributed/operate/index.adoc
@@ -6,17 +6,17 @@ Once you have a Distributed James server up and running you then need to ensure
You may also need to perform some operation maintenance or recover from incidents. This section covers
these topics.
-Read more about link:logging.adoc[Logging].
+Read more about xref:distributed/operate/logging.adoc[Logging].
-The link:webadmin.adoc[WebAdmin Restfull administration API] is the
+The xref:distributed/operate/webadmin.adoc[WebAdmin Restfull administration API] is the
recommended way to operate the Distributed James server. It allows managing and interacting with most
server components.
-The link:cli.adoc[Command line interface] allows to interact with some
+The xref:distributed/operate/cli.adoc[Command line interface] allows to interact with some
server components. However it relies on JMX technologies and its use is discouraged.
-The link:metrics.adoc[metrics] allows to build latency and throughput
+The xref:distributed/operate/metrics.adoc[metrics] allows to build latency and throughput
graphs, that can be visualized, for instance in *Grafana*.
-Finally, we did put together a link:metrics.adoc[detailed guide] for
+Finally, we did put together a xref:distributed/operate/guide.adoc[detailed guide] for
distributed James operators.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 01/31: JAMES-3296 Add republishing to
RabbitMQMailQueue from Cassandra capability
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 17f8fc5696b56c159a23d6d4abbde0d996756baf
Author: Rémi Kowalski <rk...@linagora.com>
AuthorDate: Mon Jul 6 15:16:48 2020 +0200
JAMES-3296 Add republishing to RabbitMQMailQueue from Cassandra capability
---
.../james/webadmin/dto/MailQueueItemDTOTest.java | 3 +-
.../james/queue/api/ManageableMailQueue.java | 18 +-
.../james/queue/file/FileCacheableMailQueue.java | 2 +-
.../james/queue/jms/JMSCacheableMailQueue.java | 2 +-
.../james/queue/memory/MemoryMailQueueFactory.java | 4 +-
.../org/apache/james/queue/rabbitmq/Dequeuer.java | 6 +-
.../org/apache/james/queue/rabbitmq/Enqueuer.java | 8 +
.../james/queue/rabbitmq/RabbitMQMailQueue.java | 17 +-
.../queue/rabbitmq/view/api/MailQueueView.java | 9 +-
.../view/cassandra/CassandraMailQueueBrowser.java | 64 ++++++-
.../view/cassandra/CassandraMailQueueView.java | 22 ++-
.../queue/rabbitmq/RabbitMQMailQueueTest.java | 193 ++++++++++++++++++++-
.../rabbitmq/RabbitMqMailQueueFactoryTest.java | 3 +-
13 files changed, 322 insertions(+), 29 deletions(-)
diff --git a/server/protocols/webadmin/webadmin-mailqueue/src/test/java/org/apache/james/webadmin/dto/MailQueueItemDTOTest.java b/server/protocols/webadmin/webadmin-mailqueue/src/test/java/org/apache/james/webadmin/dto/MailQueueItemDTOTest.java
index 18b14d2..4b177de 100644
--- a/server/protocols/webadmin/webadmin-mailqueue/src/test/java/org/apache/james/webadmin/dto/MailQueueItemDTOTest.java
+++ b/server/protocols/webadmin/webadmin-mailqueue/src/test/java/org/apache/james/webadmin/dto/MailQueueItemDTOTest.java
@@ -25,6 +25,7 @@ import java.util.List;
import org.apache.james.core.MailAddress;
import org.apache.james.queue.api.Mails;
+import org.apache.james.queue.api.ManageableMailQueue;
import org.apache.james.queue.api.ManageableMailQueue.MailQueueItemView;
import org.apache.mailet.base.test.FakeMail;
import org.assertj.core.api.JUnitSoftAssertions;
@@ -54,7 +55,7 @@ public class MailQueueItemDTOTest {
public void fromShouldCreateTheRightObject() throws Exception {
FakeMail mail = Mails.defaultMail().name("name").build();
ZonedDateTime date = ZonedDateTime.parse("2018-01-02T11:22:02Z");
- MailQueueItemView mailQueueItemView = new MailQueueItemView(mail, date);
+ MailQueueItemView mailQueueItemView = new ManageableMailQueue.DefaultMailQueueItemView(mail, date);
MailQueueItemDTO mailQueueItemDTO = MailQueueItemDTO.from(mailQueueItemView);
List<String> expectedRecipients = mail.getRecipients().stream()
.map(MailAddress::asString)
diff --git a/server/queue/queue-api/src/main/java/org/apache/james/queue/api/ManageableMailQueue.java b/server/queue/queue-api/src/main/java/org/apache/james/queue/api/ManageableMailQueue.java
index 20635a6..5309dbc 100644
--- a/server/queue/queue-api/src/main/java/org/apache/james/queue/api/ManageableMailQueue.java
+++ b/server/queue/queue-api/src/main/java/org/apache/james/queue/api/ManageableMailQueue.java
@@ -91,20 +91,30 @@ public interface ManageableMailQueue extends MailQueue {
/**
* Represent a View over a queue {@link MailQueue.MailQueueItem}
*/
- class MailQueueItemView {
+ interface MailQueueItemView {
+ Mail getMail();
+
+ Optional<ZonedDateTime> getNextDelivery();
+ }
+
+
+ /**
+ * Represent a View over a queue {@link MailQueue.MailQueueItem}
+ */
+ class DefaultMailQueueItemView implements MailQueueItemView {
private final Mail mail;
private final Optional<ZonedDateTime> nextDelivery;
- public MailQueueItemView(Mail mail) {
+ public DefaultMailQueueItemView(Mail mail) {
this(mail, Optional.empty());
}
- public MailQueueItemView(Mail mail, ZonedDateTime nextDelivery) {
+ public DefaultMailQueueItemView(Mail mail, ZonedDateTime nextDelivery) {
this(mail, Optional.of(nextDelivery));
}
- public MailQueueItemView(Mail mail, Optional<ZonedDateTime> nextDelivery) {
+ public DefaultMailQueueItemView(Mail mail, Optional<ZonedDateTime> nextDelivery) {
this.mail = mail;
this.nextDelivery = nextDelivery;
}
diff --git a/server/queue/queue-file/src/main/java/org/apache/james/queue/file/FileCacheableMailQueue.java b/server/queue/queue-file/src/main/java/org/apache/james/queue/file/FileCacheableMailQueue.java
index d32968e..cb69709 100644
--- a/server/queue/queue-file/src/main/java/org/apache/james/queue/file/FileCacheableMailQueue.java
+++ b/server/queue/queue-file/src/main/java/org/apache/james/queue/file/FileCacheableMailQueue.java
@@ -477,7 +477,7 @@ public class FileCacheableMailQueue implements ManageableMailQueue {
while (items.hasNext()) {
try (ObjectInputStream in = new ObjectInputStream(new FileInputStream(items.next().getObjectFile()))) {
final Mail mail = (Mail) in.readObject();
- item = new MailQueueItemView(mail, getNextDelivery(mail));
+ item = new DefaultMailQueueItemView(mail, getNextDelivery(mail));
return true;
} catch (IOException | ClassNotFoundException e) {
LOGGER.info("Unable to load mail", e);
diff --git a/server/queue/queue-jms/src/main/java/org/apache/james/queue/jms/JMSCacheableMailQueue.java b/server/queue/queue-jms/src/main/java/org/apache/james/queue/jms/JMSCacheableMailQueue.java
index 4e7cee9..1084b17 100644
--- a/server/queue/queue-jms/src/main/java/org/apache/james/queue/jms/JMSCacheableMailQueue.java
+++ b/server/queue/queue-jms/src/main/java/org/apache/james/queue/jms/JMSCacheableMailQueue.java
@@ -647,7 +647,7 @@ public class JMSCacheableMailQueue implements ManageableMailQueue, JMSSupport, M
while (hasNext()) {
try {
Message m = messages.nextElement();
- return new MailQueueItemView(createMail(m), nextDeliveryDate(m));
+ return new DefaultMailQueueItemView(createMail(m), nextDeliveryDate(m));
} catch (MessagingException | JMSException e) {
LOGGER.error("Unable to browse queue", e);
}
diff --git a/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java b/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
index da9e8b1..93f12ca 100644
--- a/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
+++ b/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
@@ -221,9 +221,9 @@ public class MemoryMailQueueFactory implements MailQueueFactory<MemoryMailQueueF
@Override
public MailQueueIterator browse() throws MailQueueException {
- Iterator<MailQueueItemView> underlying = ImmutableList.copyOf(mailItems)
+ Iterator<DefaultMailQueueItemView> underlying = ImmutableList.copyOf(mailItems)
.stream()
- .map(item -> new MailQueueItemView(item.getMail(), item.delivery))
+ .map(item -> new DefaultMailQueueItemView(item.getMail(), item.delivery))
.iterator();
return new MailQueueIterator() {
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Dequeuer.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Dequeuer.java
index fe209f9..b477335 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Dequeuer.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Dequeuer.java
@@ -32,6 +32,7 @@ import org.apache.james.queue.api.MailQueue;
import org.apache.james.queue.api.MailQueueFactory;
import org.apache.james.queue.rabbitmq.view.api.DeleteCondition;
import org.apache.james.queue.rabbitmq.view.api.MailQueueView;
+import org.apache.james.queue.rabbitmq.view.cassandra.CassandraMailQueueBrowser;
import org.apache.mailet.Mail;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -81,13 +82,13 @@ class Dequeuer implements Closeable {
private final MailLoader mailLoader;
private final Metric dequeueMetric;
private final MailReferenceSerializer mailReferenceSerializer;
- private final MailQueueView mailQueueView;
+ private final MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> mailQueueView;
private final Receiver receiver;
private final Flux<AcknowledgableDelivery> flux;
Dequeuer(MailQueueName name, ReceiverProvider receiverProvider, MailLoader mailLoader,
MailReferenceSerializer serializer, MetricFactory metricFactory,
- MailQueueView mailQueueView, MailQueueFactory.PrefetchCount prefetchCount) {
+ MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> mailQueueView, MailQueueFactory.PrefetchCount prefetchCount) {
this.mailLoader = mailLoader;
this.mailReferenceSerializer = serializer;
this.mailQueueView = mailQueueView;
@@ -162,5 +163,4 @@ class Dequeuer implements Closeable {
return Mono.empty();
});
}
-
}
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Enqueuer.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Enqueuer.java
index c134798..af30c14 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Enqueuer.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/Enqueuer.java
@@ -34,6 +34,7 @@ import org.apache.james.metrics.api.Metric;
import org.apache.james.metrics.api.MetricFactory;
import org.apache.james.queue.api.MailQueue;
import org.apache.james.queue.rabbitmq.view.api.MailQueueView;
+import org.apache.james.queue.rabbitmq.view.cassandra.CassandraMailQueueBrowser;
import org.apache.mailet.Mail;
import com.fasterxml.jackson.core.JsonProcessingException;
@@ -76,6 +77,13 @@ class Enqueuer {
.block();
}
+ Mono<Void> reQueue(CassandraMailQueueBrowser.CassandraMailQueueItemView item) {
+ Mail mail = item.getMail();
+ return Mono.fromCallable(() -> new MailReference(item.getEnqueuedId(), mail, item.getEnqueuedPartsId()))
+ .flatMap(Throwing.function(this::publishReferenceToRabbit).sneakyThrow())
+ .then();
+ }
+
private Mono<MimeMessagePartsId> saveMail(Mail mail) throws MailQueue.MailQueueException {
try {
return mimeMessageStore.save(mail.getMessage());
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueue.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueue.java
index 3ed3ba9..5e838db 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueue.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueue.java
@@ -20,20 +20,24 @@
package org.apache.james.queue.rabbitmq;
import java.time.Duration;
+import java.time.Instant;
import org.apache.james.metrics.api.MetricFactory;
import org.apache.james.queue.api.MailQueueItemDecoratorFactory;
import org.apache.james.queue.api.ManageableMailQueue;
import org.apache.james.queue.rabbitmq.view.api.DeleteCondition;
import org.apache.james.queue.rabbitmq.view.api.MailQueueView;
+import org.apache.james.queue.rabbitmq.view.cassandra.CassandraMailQueueBrowser;
import org.apache.mailet.Mail;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.github.fge.lambdas.Throwing;
+import com.google.common.base.Function;
import com.google.common.base.MoreObjects;
import reactor.core.publisher.Flux;
+import reactor.core.publisher.Mono;
public class RabbitMQMailQueue implements ManageableMailQueue {
@@ -43,12 +47,12 @@ public class RabbitMQMailQueue implements ManageableMailQueue {
private final MetricFactory metricFactory;
private final Enqueuer enqueuer;
private final Dequeuer dequeuer;
- private final MailQueueView mailQueueView;
+ private final MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> mailQueueView;
private final MailQueueItemDecoratorFactory decoratorFactory;
RabbitMQMailQueue(MetricFactory metricFactory, MailQueueName name,
Enqueuer enqueuer, Dequeuer dequeuer,
- MailQueueView mailQueueView, MailQueueItemDecoratorFactory decoratorFactory) {
+ MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> mailQueueView, MailQueueItemDecoratorFactory decoratorFactory) {
this.metricFactory = metricFactory;
this.name = name;
this.enqueuer = enqueuer;
@@ -119,4 +123,13 @@ public class RabbitMQMailQueue implements ManageableMailQueue {
.add("name", name)
.toString();
}
+
+ public Flux<String> republishNotProcessedMails(Instant olderThan) {
+ Function<CassandraMailQueueBrowser.CassandraMailQueueItemView, Mono<String>> requeue = item ->
+ enqueuer.reQueue(item)
+ .thenReturn(item.getMail().getName());
+
+ return mailQueueView.browseOlderThanReactive(olderThan)
+ .flatMap(requeue);
+ }
}
\ No newline at end of file
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/api/MailQueueView.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/api/MailQueueView.java
index 33eec80..921a08b 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/api/MailQueueView.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/api/MailQueueView.java
@@ -19,14 +19,17 @@
package org.apache.james.queue.rabbitmq.view.api;
+import java.time.Instant;
+
import org.apache.james.queue.api.ManageableMailQueue;
import org.apache.james.queue.rabbitmq.EnqueueId;
import org.apache.james.queue.rabbitmq.EnqueuedItem;
import org.apache.james.queue.rabbitmq.MailQueueName;
+import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
-public interface MailQueueView {
+public interface MailQueueView<V extends ManageableMailQueue.MailQueueItemView> {
interface Factory {
MailQueueView create(MailQueueName mailQueueName);
@@ -42,5 +45,9 @@ public interface MailQueueView {
ManageableMailQueue.MailQueueIterator browse();
+ Flux<V> browseReactive();
+
+ Flux<V> browseOlderThanReactive(Instant olderThan);
+
long getSize();
}
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueBrowser.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueBrowser.java
index 91223ad..455025c 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueBrowser.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueBrowser.java
@@ -24,17 +24,21 @@ import static org.apache.james.queue.rabbitmq.view.cassandra.model.BucketedSlice
import java.time.Clock;
import java.time.Instant;
+import java.time.ZonedDateTime;
import java.util.Comparator;
import java.util.Iterator;
+import java.util.Optional;
import javax.inject.Inject;
import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.james.blob.api.Store;
import org.apache.james.blob.mail.MimeMessagePartsId;
import org.apache.james.blob.mail.MimeMessageStore;
import org.apache.james.queue.api.ManageableMailQueue;
+import org.apache.james.queue.rabbitmq.EnqueueId;
import org.apache.james.queue.rabbitmq.EnqueuedItem;
import org.apache.james.queue.rabbitmq.MailQueueName;
import org.apache.james.queue.rabbitmq.view.cassandra.configuration.CassandraMailQueueViewConfiguration;
@@ -52,9 +56,9 @@ public class CassandraMailQueueBrowser {
static class CassandraMailQueueIterator implements ManageableMailQueue.MailQueueIterator {
- private final Iterator<ManageableMailQueue.MailQueueItemView> iterator;
+ private final Iterator<CassandraMailQueueItemView> iterator;
- CassandraMailQueueIterator(Iterator<ManageableMailQueue.MailQueueItemView> iterator) {
+ CassandraMailQueueIterator(Iterator<CassandraMailQueueItemView> iterator) {
Preconditions.checkNotNull(iterator);
this.iterator = iterator;
@@ -71,7 +75,7 @@ public class CassandraMailQueueBrowser {
}
@Override
- public ManageableMailQueue.MailQueueItemView next() {
+ public CassandraMailQueueItemView next() {
return iterator.next();
}
}
@@ -100,10 +104,24 @@ public class CassandraMailQueueBrowser {
this.clock = clock;
}
- Flux<ManageableMailQueue.MailQueueItemView> browse(MailQueueName queueName) {
+ Flux<CassandraMailQueueItemView> browse(MailQueueName queueName) {
return browseReferences(queueName)
.flatMapSequential(this::toMailFuture)
- .map(ManageableMailQueue.MailQueueItemView::new);
+ .map(CassandraMailQueueItemView::new);
+ }
+
+ Flux<CassandraMailQueueItemView> browseOlderThan(MailQueueName queueName, Instant olderThan) {
+ return browseReferencesOlderThan(queueName, olderThan)
+ .flatMapSequential(this::toMailFuture)
+ .map(CassandraMailQueueItemView::new);
+ }
+
+ Flux<EnqueuedItemWithSlicingContext> browseReferencesOlderThan(MailQueueName queueName, Instant olderThan) {
+ return browseStartDao.findBrowseStart(queueName)
+ .flatMapMany(this::allSlicesStartingAt)
+ .filter(slice -> slice.getStartSliceInstant().isBefore(olderThan))
+ .flatMapSequential(slice -> browseSlice(queueName, slice))
+ .filter(item -> item.getEnqueuedItem().getEnqueuedTime().isBefore(olderThan));
}
Flux<EnqueuedItemWithSlicingContext> browseReferences(MailQueueName queueName) {
@@ -112,10 +130,10 @@ public class CassandraMailQueueBrowser {
.flatMapSequential(slice -> browseSlice(queueName, slice));
}
- private Mono<Mail> toMailFuture(EnqueuedItemWithSlicingContext enqueuedItemWithSlicingContext) {
+ private Mono<Pair<EnqueuedItem, Mail>> toMailFuture(EnqueuedItemWithSlicingContext enqueuedItemWithSlicingContext) {
EnqueuedItem enqueuedItem = enqueuedItemWithSlicingContext.getEnqueuedItem();
return mimeMessageStore.read(enqueuedItem.getPartsId())
- .map(mimeMessage -> toMail(enqueuedItem, mimeMessage));
+ .map(mimeMessage -> Pair.of(enqueuedItem, toMail(enqueuedItem, mimeMessage)));
}
private Mail toMail(EnqueuedItem enqueuedItem, MimeMessage mimeMessage) {
@@ -151,4 +169,36 @@ public class CassandraMailQueueBrowser {
.range(0, configuration.getBucketCount())
.map(BucketId::of);
}
+
+ public static class CassandraMailQueueItemView implements ManageableMailQueue.MailQueueItemView {
+ private final EnqueuedItem enqueuedItem;
+ private final Mail mail;
+
+ public CassandraMailQueueItemView(Pair<EnqueuedItem, Mail> pair) {
+ this(pair.getLeft(), pair.getRight());
+ }
+
+ public CassandraMailQueueItemView(EnqueuedItem enqueuedItem, Mail mail) {
+ this.enqueuedItem = enqueuedItem;
+ this.mail = mail;
+ }
+
+ public EnqueueId getEnqueuedId() {
+ return enqueuedItem.getEnqueueId();
+ }
+
+ public MimeMessagePartsId getEnqueuedPartsId() {
+ return enqueuedItem.getPartsId();
+ }
+
+ @Override
+ public Mail getMail() {
+ return mail;
+ }
+
+ @Override
+ public Optional<ZonedDateTime> getNextDelivery() {
+ return Optional.empty();
+ }
+ }
}
diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueView.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueView.java
index b82de97..72b26a3 100644
--- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueView.java
+++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/view/cassandra/CassandraMailQueueView.java
@@ -21,6 +21,8 @@ package org.apache.james.queue.rabbitmq.view.cassandra;
import static org.apache.james.util.FunctionalUtils.negate;
+import java.time.Instant;
+
import javax.inject.Inject;
import org.apache.james.queue.api.ManageableMailQueue;
@@ -33,10 +35,11 @@ import org.apache.james.queue.rabbitmq.view.cassandra.configuration.CassandraMai
import org.apache.james.queue.rabbitmq.view.cassandra.configuration.EventsourcingConfigurationManagement;
import org.apache.james.queue.rabbitmq.view.cassandra.model.EnqueuedItemWithSlicingContext;
+import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
-public class CassandraMailQueueView implements MailQueueView {
+public class CassandraMailQueueView implements MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> {
public static class Factory implements MailQueueView.Factory {
private final CassandraMailQueueMailStore storeHelper;
@@ -91,13 +94,24 @@ public class CassandraMailQueueView implements MailQueueView {
@Override
public ManageableMailQueue.MailQueueIterator browse() {
return new CassandraMailQueueBrowser.CassandraMailQueueIterator(
- cassandraMailQueueBrowser.browse(mailQueueName)
- .subscribeOn(Schedulers.elastic())
+ browseReactive()
.toIterable()
.iterator());
}
@Override
+ public Flux<CassandraMailQueueBrowser.CassandraMailQueueItemView> browseReactive() {
+ return cassandraMailQueueBrowser.browse(mailQueueName)
+ .subscribeOn(Schedulers.elastic());
+ }
+
+ @Override
+ public Flux<CassandraMailQueueBrowser.CassandraMailQueueItemView> browseOlderThanReactive(Instant olderThan) {
+ return cassandraMailQueueBrowser.browseOlderThan(mailQueueName, olderThan)
+ .subscribeOn(Schedulers.elastic());
+ }
+
+ @Override
public long getSize() {
return cassandraMailQueueBrowser.browseReferences(mailQueueName)
.count()
@@ -133,6 +147,6 @@ public class CassandraMailQueueView implements MailQueueView {
@Override
public Mono<Boolean> isPresent(EnqueueId id) {
return cassandraMailQueueMailDelete.isDeleted(id, mailQueueName)
- .map(negate());
+ .map(negate());
}
}
diff --git a/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueueTest.java b/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueueTest.java
index bf2d209..8808bfc 100644
--- a/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueueTest.java
+++ b/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMQMailQueueTest.java
@@ -23,6 +23,7 @@ import static java.time.temporal.ChronoUnit.HOURS;
import static org.apache.james.backends.cassandra.Scenario.Builder.executeNormally;
import static org.apache.james.backends.cassandra.Scenario.Builder.fail;
import static org.apache.james.backends.cassandra.Scenario.Builder.returnEmpty;
+import static org.apache.james.backends.rabbitmq.Constants.EMPTY_ROUTING_KEY;
import static org.apache.james.queue.api.Mails.defaultMail;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
@@ -75,11 +76,12 @@ import org.junit.jupiter.api.extension.RegisterExtension;
import org.mockito.ArgumentCaptor;
import com.github.fge.lambdas.Throwing;
-
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
+import reactor.rabbitmq.BindingSpecification;
import reactor.rabbitmq.OutboundMessage;
+import reactor.rabbitmq.Sender;
class RabbitMQMailQueueTest {
private static final HashBlobId.Factory BLOB_ID_FACTORY = new HashBlobId.Factory();
@@ -132,7 +134,7 @@ class RabbitMQMailQueueTest {
}
@Override
- public MailQueue getMailQueue() {
+ public RabbitMQMailQueue getMailQueue() {
return mailQueue;
}
@@ -288,6 +290,177 @@ class RabbitMQMailQueueTest {
.containsExactly(name1, name2, name3);
}
+ @Test
+ void messagesShouldBeProcessedAfterNotPublishedMailsHaveBeenReprocessed() throws Exception {
+ clock.setInstant(Instant.now().minus(Duration.ofHours(2)));
+ String name1 = "myMail1";
+ String name2 = "myMail2";
+ String name3 = "myMail3";
+ Flux<MailQueue.MailQueueItem> dequeueFlux = Flux.from(getMailQueue().deQueue());
+
+ // Avoid early processing and prefetching
+ Sender sender = rabbitMQExtension.getSender();
+
+ suspendDequeuing(sender);
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name1)
+ .build());
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name2)
+ .build());
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name3)
+ .build());
+
+ resumeDequeuing(sender);
+ assertThat(getMailQueue()
+ .republishNotProcessedMails(Instant.now().minus(Duration.ofHours(1)))
+ .collectList()
+ .block())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+
+ List<MailQueue.MailQueueItem> items = dequeueFlux.take(Duration.ofSeconds(10)).collectList().block();
+
+ assertThat(items)
+ .extracting(item -> item.getMail().getName())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+ }
+
+ @Test
+ void onlyOldMessagesShouldBeProcessedAfterNotPublishedMailsHaveBeenReprocessed() throws Exception {
+ clock.setInstant(Instant.now().minus(Duration.ofHours(2)));
+ String name1 = "myMail1";
+ String name2 = "myMail2";
+ String name3 = "myMail3";
+ Flux<MailQueue.MailQueueItem> dequeueFlux = Flux.from(getMailQueue().deQueue());
+
+ // Avoid early processing and prefetching
+ Sender sender = rabbitMQExtension.getSender();
+
+ suspendDequeuing(sender);
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name1)
+ .build());
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name2)
+ .build());
+
+ clock.setInstant(Instant.now());
+ getMailQueue().enQueue(defaultMail()
+ .name(name3)
+ .build());
+
+ resumeDequeuing(sender);
+ assertThat(getMailQueue()
+ .republishNotProcessedMails(Instant.now().minus(Duration.ofHours(1)))
+ .collectList()
+ .block())
+ .containsExactlyInAnyOrder(name1, name2);
+
+ List<MailQueue.MailQueueItem> items = dequeueFlux.take(Duration.ofSeconds(10)).collectList().block();
+
+ assertThat(items)
+ .extracting(item -> item.getMail().getName())
+ .containsExactlyInAnyOrder(name1, name2);
+ }
+
+ @Test
+ void messagesShouldBeProcessedAfterTwoMailsReprocessing() throws Exception {
+ clock.setInstant(Instant.now().minus(Duration.ofHours(2)));
+ String name1 = "myMail1";
+ String name2 = "myMail2";
+ String name3 = "myMail3";
+ Flux<MailQueue.MailQueueItem> dequeueFlux = Flux.from(getMailQueue().deQueue());
+
+ // Avoid early processing and prefetching
+ Sender sender = rabbitMQExtension.getSender();
+
+ suspendDequeuing(sender);
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name1)
+ .build());
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name2)
+ .build());
+
+ getMailQueue().enQueue(defaultMail()
+ .name(name3)
+ .build());
+
+ assertThat(getMailQueue()
+ .republishNotProcessedMails(Instant.now().minus(Duration.ofHours(1)))
+ .collectList()
+ .block())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+ resumeDequeuing(sender);
+ assertThat(getMailQueue()
+ .republishNotProcessedMails(Instant.now().minus(Duration.ofHours(1)))
+ .collectList()
+ .block())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+
+ List<MailQueue.MailQueueItem> items = dequeueFlux.take(Duration.ofSeconds(10)).collectList().block();
+
+ assertThat(items)
+ .extracting(item -> item.getMail().getName())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+ }
+
+ @Test
+ void messagesShouldBeProcessedAfterNotPublishedMailsHaveBeenReprocessedAndNewMessagesShouldNotBeLost() throws Exception {
+ clock.setInstant(Instant.now().minus(Duration.ofHours(2)));
+ String name1 = "myMail1";
+ String name2 = "myMail2";
+ String name3 = "myMail3";
+ Flux<MailQueue.MailQueueItem> dequeueFlux = Flux.from(getMailQueue().deQueue());
+
+ // Avoid early processing and prefetching
+ Sender sender = rabbitMQExtension.getSender();
+
+ suspendDequeuing(sender);
+ //mail send when rabbit down
+ getMailQueue().enQueue(defaultMail()
+ .name(name1)
+ .build());
+ resumeDequeuing(sender);
+
+ //mail send when rabbit is up again and before rebuild
+ clock.setInstant(Instant.now());
+ getMailQueue().enQueue(defaultMail()
+ .name(name3)
+ .build());
+
+ Flux.merge(Mono.fromCallable(() -> {
+ //mail send concurently with rebuild
+ getMailQueue().enQueue(defaultMail()
+ .name(name2)
+ .build());
+ return true;
+
+ }), Mono.fromRunnable(() ->
+ assertThat(getMailQueue()
+ .republishNotProcessedMails(Instant.now().minus(Duration.ofHours(1)))
+ .collectList()
+ .block())
+ .containsOnly(name1)
+ ))
+ .then()
+ .block(Duration.ofSeconds(10));
+
+ List<MailQueue.MailQueueItem> items = dequeueFlux.take(Duration.ofSeconds(10)).collectList().block();
+
+ assertThat(items)
+ .extracting(item -> item.getMail().getName())
+ .containsExactlyInAnyOrder(name1, name2, name3);
+ }
+
private void enqueueSomeMails(Function<Integer, String> namePattern, int emailCount) {
IntStream.rangeClosed(1, emailCount)
.forEach(Throwing.intConsumer(i -> enQueue(defaultMail()
@@ -485,6 +658,22 @@ class RabbitMQMailQueueTest {
Awaitility.await().atMost(org.awaitility.Duration.TEN_SECONDS)
.untilAsserted(() -> assertThat(deadLetteredCount.get()).isEqualTo(1));
}
+
+ private void resumeDequeuing(Sender sender) {
+ sender.bindQueue(getMailQueueBindingSpecification()).block();
+ }
+
+ private void suspendDequeuing(Sender sender) {
+ sender.unbindQueue(getMailQueueBindingSpecification()).block();
+ }
+
+ private BindingSpecification getMailQueueBindingSpecification() {
+ MailQueueName mailQueueName = MailQueueName.fromString(getMailQueue().getName().asString());
+ return BindingSpecification.binding()
+ .exchange(mailQueueName.toRabbitExchangeName().asString())
+ .queue(mailQueueName.toWorkQueueName().asString())
+ .routingKey(EMPTY_ROUTING_KEY);
+ }
}
@Nested
diff --git a/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMqMailQueueFactoryTest.java b/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMqMailQueueFactoryTest.java
index c40fb11..2917b35 100644
--- a/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMqMailQueueFactoryTest.java
+++ b/server/queue/queue-rabbitmq/src/test/java/org/apache/james/queue/rabbitmq/RabbitMqMailQueueFactoryTest.java
@@ -35,6 +35,7 @@ import org.apache.james.queue.api.MailQueueFactoryContract;
import org.apache.james.queue.api.RawMailQueueItemDecoratorFactory;
import org.apache.james.queue.rabbitmq.view.RabbitMQMailQueueConfiguration;
import org.apache.james.queue.rabbitmq.view.api.MailQueueView;
+import org.apache.james.queue.rabbitmq.view.cassandra.CassandraMailQueueBrowser;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -52,7 +53,7 @@ class RabbitMqMailQueueFactoryTest implements MailQueueFactoryContract<RabbitMQM
void setup() throws Exception {
MimeMessageStore.Factory mimeMessageStoreFactory = mock(MimeMessageStore.Factory.class);
MailQueueView.Factory mailQueueViewFactory = mock(MailQueueView.Factory.class);
- MailQueueView mailQueueView = mock(MailQueueView.class);
+ MailQueueView<CassandraMailQueueBrowser.CassandraMailQueueItemView> mailQueueView = mock(MailQueueView.class);
when(mailQueueViewFactory.create(any()))
.thenReturn(mailQueueView);
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 20/31: JAMES-3302 Migrate Run with docker section
for Distributed server
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 96c4f1d0ee77b04a3a07c73e377e6fa63263a8a2
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 11:52:28 2020 +0700
JAMES-3302 Migrate Run with docker section for Distributed server
---
.../servers/pages/distributed/run-docker.adoc | 90 +++++++++++++++++++++-
1 file changed, 88 insertions(+), 2 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/run-docker.adoc b/docs/modules/servers/pages/distributed/run-docker.adoc
index 806e6dd..d3fd897 100644
--- a/docs/modules/servers/pages/distributed/run-docker.adoc
+++ b/docs/modules/servers/pages/distributed/run-docker.adoc
@@ -1,4 +1,90 @@
= Run with docker
-(TODO adapt content from
-https://github.com/linagora/james-project#run-james-with-guice--cassandra--rabbitmq--swift--elasticsearch)
\ No newline at end of file
+== Requirements
+
+Built artifacts should be in ./dockerfiles/run/guice/cassandra-rabbitmq/destination folder for cassandra.
+If you haven't already:
+
+ $ docker build -t james/project dockerfiles/compilation/java-11
+ $ docker run -v $HOME/.m2:/root/.m2 -v $PWD:/origin \
+ -v $PWD/dockerfiles/run/guice/cassandra-rabbitmq/destination:/cassandra-rabbitmq/destination \
+ -t james/project -s HEAD
+
+== Running
+
+You need a running *cassandra* in docker. To achieve this run:
+
+ $ docker run -d --name=cassandra cassandra:3.11.3
+
+You need a running *rabbitmq* in docker. To achieve this run:
+
+ $ docker run -d --name=rabbitmq rabbitmq:3.8.1-management
+
+You need a running *swift* objectstorage in docker. To achieve this run:
+
+ $ docker run -d --name=swift linagora/openstack-keystone-swift:pike
+
+You need a running *ElasticSearch* in docker. To achieve this run:
+
+ $ docker run -d --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:6.3.2
+
+If you want to use all the JMAP search capabilities, you may also need to start Tika:
+
+ $ docker run -d --name=tika apache/tika:1.24
+
+You can find more explanation on the need of Tika in this page http://james.apache.org/server/config-elasticsearch.html
+
+We need to provide the key we will use for TLS. For obvious reasons, this is not provided in this git.
+
+Copy your TLS keys to `run/guice/cassandra-rabbitmq/destination/conf/keystore` or generate it using the following command. The password must be `james72laBalle` to match default configuration.
+
+ $ keytool -genkey -alias james -keyalg RSA -keystore dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/keystore
+
+Then we need to build james container :
+
+ $ docker build -t james_run dockerfiles/run/guice/cassandra-rabbitmq
+
+To run this container :
+
+ $ docker run --hostname HOSTNAME -p "25:25" -p 80:80 -p "110:110" -p "143:143" -p "465:465" -p "587:587" -p "993:993" -p "127.0.0.1:8000:8000" --link cassandra:cassandra --link rabbitmq:rabbitmq
+ --link elasticsearch:elasticsearch --link tika:tika --link swift:swift --name james_run -t james_run
+
+Where :
+
+- HOSTNAME: is the hostname you want to give to your James container. This DNS entry will be used to send mail to your James server.
+
+Webadmin port binding is restricted to loopback as users are not authenticated by default on webadmin server. Thus you should avoid exposing it in production.
+Note that the above example assumes `127.0.0.1` is your loopback interface for convenience but you should change it if this is not the case on your machine.
+
+If you want to pass additional options to the underlying java command, you can configure a _JVM_OPTIONS_ env variable, for example add:
+
+ --env JVM_OPTIONS="-Xms256m -Xmx2048m"
+
+To have log file accessible on a volume, add *-v $PWD/logs:/logs* option to the above command line, where *$PWD/logs* is your local directory to put files in.
+
+== Instrumentation
+You can use Glowroot to instrumentalize James. The provided guice docker files allow a simple way to do it.
+In order to activate Glowroot you need to run the container with the environment variable _GLOWROOT_ACTIVATED_ set to _true_
+and to expose the glowroot instrumentation ui port.
+
+ --env GLOWROOT_ACTIVATED=true -p "4000:4000"
+
+By default, the Glowroot UI is accessible from every machines in the network as defined in the _destination/admin.json_.
+Which you could configure before building the image, if you want to restrict its accessibility to localhost for example.
+See the https://github.com/glowroot/glowroot/wiki/Agent-Installation-(with-Embedded-Collector)#user-content-optional-post-installation-steps[Glowroot post installation steps] for more details.
+
+Or by mapping the 4000 port to the IP of the desired network interface, for example `-p 127.0.0.1:4000:4000`.
+
+
+== Handling attachment indexing
+
+You can handle attachment text extraction before indexing in ElasticSearch. This makes attachments searchable. To enable this:
+
+Run tika:
+
+ $ docker run --name tika apache/tika:1.24
+
+Add a link for the tika container in the above command line:
+
+ $ docker run --hostname HOSTNAME -p "25:25" -p 80:80 -p "110:110" -p "143:143" -p "465:465" -p "587:587" -p "993:993" --link cassandra:cassandra --link rabbitmq:rabbitmq
+ --link elasticsearch:elasticsearch --link tika:tika --name james_run -t james_run
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 15/31: [REFACTORING] IMAP FETCH javaDoc fixes
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 3f1f17194c30e0ffffa60e16a8529a20b9dc8f9b
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:39:27 2020 +0700
[REFACTORING] IMAP FETCH javaDoc fixes
This solves some intellij warnings
---
.../java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java | 5 +----
.../org/apache/james/imap/processor/fetch/HeadersBodyElement.java | 1 -
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
index 123c152..7513443 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
@@ -97,11 +97,8 @@ public final class EnvelopeBuilder {
/**
* Try to parse the addresses out of the header. If its not possible because
* of a {@link ParseException} a null value is returned
- *
- * @param message
- * @param headerName
+ *
* @return addresses
- * @throws MailboxException
*/
private FetchResponse.Envelope.Address[] buildAddresses(Headers message, String headerName) throws MailboxException {
final Header header = MessageResultUtils.getMatching(headerName, message.headers());
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
index f11375e..9b81c2e 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/HeadersBodyElement.java
@@ -35,7 +35,6 @@ public class HeadersBodyElement extends ContentBodyElement {
/**
* Indicate that there is no text body in the message. In this case we don't need to write a single CRLF in anycase if
* this Element does not contain a header.
- * @throws IOException
*/
public void noBody() throws IOException {
if (super.size() == 0) {
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 04/31: JAMES-3305 Add FailsDeserializationTask type
for testing
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7ed19807ad79e3899097469152457865889d302e
Author: LanKhuat <dl...@linagora.com>
AuthorDate: Wed Jul 15 17:21:21 2020 +0700
JAMES-3305 Add FailsDeserializationTask type for testing
---
.../james/task/FailsDeserializationTask.java | 40 ++++++++++++++++++++++
.../task/json/dto/FailsDeserializationTaskDTO.java | 35 +++++++++++++++++++
.../server/task/json/dto/TestTaskDTOModules.java | 9 +++++
3 files changed, 84 insertions(+)
diff --git a/server/task/task-api/src/test/java/org/apache/james/task/FailsDeserializationTask.java b/server/task/task-api/src/test/java/org/apache/james/task/FailsDeserializationTask.java
new file mode 100644
index 0000000..9a047a4
--- /dev/null
+++ b/server/task/task-api/src/test/java/org/apache/james/task/FailsDeserializationTask.java
@@ -0,0 +1,40 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.task;
+
+import java.util.Optional;
+
+public class FailsDeserializationTask implements Task {
+ public static final TaskType TYPE = TaskType.of("fails-deserialization");
+
+ @Override
+ public Result run() {
+ return Result.COMPLETED;
+ }
+
+ @Override
+ public TaskType type() {
+ return TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.empty();
+ }
+}
diff --git a/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/FailsDeserializationTaskDTO.java b/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/FailsDeserializationTaskDTO.java
new file mode 100644
index 0000000..e043e5e
--- /dev/null
+++ b/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/FailsDeserializationTaskDTO.java
@@ -0,0 +1,35 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.server.task.json.dto;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class FailsDeserializationTaskDTO implements TaskDTO {
+
+ private final String type;
+
+ public FailsDeserializationTaskDTO(@JsonProperty("type") String type) {
+ this.type = type;
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+}
diff --git a/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/TestTaskDTOModules.java b/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/TestTaskDTOModules.java
index 61b3acc..a4450d0 100644
--- a/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/TestTaskDTOModules.java
+++ b/server/task/task-json/src/test/java/org/apache/james/server/task/json/dto/TestTaskDTOModules.java
@@ -25,6 +25,7 @@ import org.apache.james.json.DTOModule;
import org.apache.james.server.task.json.TestTask;
import org.apache.james.task.CompletedTask;
import org.apache.james.task.FailedTask;
+import org.apache.james.task.FailsDeserializationTask;
import org.apache.james.task.MemoryReferenceTask;
import org.apache.james.task.MemoryReferenceWithCounterTask;
import org.apache.james.task.ThrowingTask;
@@ -57,6 +58,14 @@ public interface TestTaskDTOModules {
.typeName("completed-task")
.withFactory(TaskDTOModule::new);
+ TaskDTOModule<FailsDeserializationTask, FailsDeserializationTaskDTO> FAILS_DESERIALIZATION_TASK_MODULE = DTOModule
+ .forDomainObject(FailsDeserializationTask.class)
+ .convertToDTO(FailsDeserializationTaskDTO.class)
+ .toDomainObjectConverter(dto -> {throw new RuntimeException("fail to deserialize"); })
+ .toDTOConverter((task, typeName) -> new FailsDeserializationTaskDTO(typeName))
+ .typeName(FailsDeserializationTask.TASK_TYPE)
+ .withFactory(TaskDTOModule::new);
+
TaskDTOModule<ThrowingTask, ThrowingTaskDTO> THROWING_TASK_MODULE = DTOModule
.forDomainObject(ThrowingTask.class)
.convertToDTO(ThrowingTaskDTO.class)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 13/31: [REFACTORING] Avoid variable reallocation in
EnvelopeBuilder
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 823ed4e84382468bf7daf1e95bb41bedc4442cdb
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:32:58 2020 +0700
[REFACTORING] Avoid variable reallocation in EnvelopeBuilder
---
.../imap/processor/fetch/EnvelopeBuilder.java | 43 ++++++++++------------
1 file changed, 19 insertions(+), 24 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
index 06bde4f..123c152 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/EnvelopeBuilder.java
@@ -63,13 +63,12 @@ public final class EnvelopeBuilder {
private String headerValue(Headers message, String headerName) throws MailboxException {
final Header header = MessageResultUtils.getMatching(headerName, message.headers());
- final String result;
if (header == null) {
- result = null;
+ return null;
} else {
final String value = header.getValue();
if (value == null || "".equals(value)) {
- result = null;
+ return null;
} else {
// ENVELOPE header values must be unfolded
@@ -80,22 +79,19 @@ public final class EnvelopeBuilder {
// unfold does. mime4j's unfold does strictly follow the rfc and so preserve them
//
// See IMAP-327 and https://mailman2.u.washington.edu/mailman/htdig/imap-protocol/2010-July/001271.html
- result = MimeUtility.unfold(value);
+ return MimeUtility.unfold(value);
}
}
- return result;
}
private FetchResponse.Envelope.Address[] buildAddresses(Headers message, String headerName, FetchResponse.Envelope.Address[] defaults) throws MailboxException {
- final FetchResponse.Envelope.Address[] results;
final FetchResponse.Envelope.Address[] addresses = buildAddresses(message, headerName);
if (addresses == null) {
- results = defaults;
+ return defaults;
} else {
- results = addresses;
+ return addresses;
}
- return results;
}
/**
@@ -109,9 +105,8 @@ public final class EnvelopeBuilder {
*/
private FetchResponse.Envelope.Address[] buildAddresses(Headers message, String headerName) throws MailboxException {
final Header header = MessageResultUtils.getMatching(headerName, message.headers());
- FetchResponse.Envelope.Address[] results;
if (header == null) {
- results = null;
+ return null;
} else {
// We need to unfold the header line.
@@ -124,9 +119,8 @@ public final class EnvelopeBuilder {
String value = MimeUtility.unfold(header.getValue());
if ("".equals(value.trim())) {
- results = null;
+ return null;
} else {
-
AddressList addressList = LenientAddressParser.DEFAULT.parseAddressList(value);
final int size = addressList.size();
final List<FetchResponse.Envelope.Address> addresses = new ArrayList<>(size);
@@ -145,22 +139,13 @@ public final class EnvelopeBuilder {
}
}
- results = addresses.toArray(FetchResponse.Envelope.Address[]::new);
-
-
+ return addresses.toArray(FetchResponse.Envelope.Address[]::new);
}
}
- return results;
}
private FetchResponse.Envelope.Address buildMailboxAddress(org.apache.james.mime4j.dom.address.Mailbox mailbox) {
- // Encode the mailbox name
- // See IMAP-266
- String name = mailbox.getName();
- if (name != null) {
- name = EncoderUtil.encodeAddressDisplayName(name);
- }
-
+ final String name = encodedMailboxName(mailbox);
final String domain = mailbox.getDomain();
final DomainList route = mailbox.getRoute();
final String atDomainList;
@@ -173,6 +158,16 @@ public final class EnvelopeBuilder {
return buildMailboxAddress(name, atDomainList, localPart, domain);
}
+ private String encodedMailboxName(Mailbox mailbox) {
+ // Encode the mailbox name
+ // See IMAP-266
+ String name = mailbox.getName();
+ if (name != null) {
+ return EncoderUtil.encodeAddressDisplayName(name);
+ }
+ return null;
+ }
+
private void addAddresses(Group group, List<FetchResponse.Envelope.Address> addresses) {
final String groupName = group.getName();
final FetchResponse.Envelope.Address start = startGroup(groupName);
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 24/31: JAMES-3302 Refine Distributed server
architecture page
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1a15f32da51c12929a79e15d9c88da83d231171e
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 14:08:38 2020 +0700
JAMES-3302 Refine Distributed server architecture page
---
docs/modules/servers/pages/distributed/architecture.adoc | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/docs/modules/servers/pages/distributed/architecture.adoc b/docs/modules/servers/pages/distributed/architecture.adoc
index 73de370..3113be9 100644
--- a/docs/modules/servers/pages/distributed/architecture.adoc
+++ b/docs/modules/servers/pages/distributed/architecture.adoc
@@ -2,6 +2,8 @@
This sections presents the Distributed Server architecture.
+== Storage
+
In order to deliver its promises, the Distributed Server leverages the following storage strategies:
(TODO picture)
@@ -14,3 +16,7 @@ In order to deliver its promises, the Distributed Server leverages the following
* *RabbitMQ* enables James nodes of a same cluster to collaborate together.
* *Tika* (optional) enables text extraction from attachments, thus improving full text search results.
* *SpamAssassin* (optional) can be used for Spam detection and user feedback is supported.
+
+== Components
+
+(TODO)
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 08/31: [REFACTORING] Avoid variable reallocation in
FetchResponseBuilder
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 06431cd7b0a9fed0c75e52ea04b757ae08854b55
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 15:45:25 2020 +0700
[REFACTORING] Avoid variable reallocation in FetchResponseBuilder
---
.../imap/processor/fetch/FetchResponseBuilder.java | 55 +++++++++-------------
1 file changed, 21 insertions(+), 34 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
index 73f18f7..4d83116 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchResponseBuilder.java
@@ -257,7 +257,6 @@ public final class FetchResponseBuilder {
}
private FetchResponse.BodyElement bodyFetch(MessageResult messageResult, BodyFetchElement fetchElement) throws MailboxException {
-
final Long firstOctet = fetchElement.getFirstOctet();
final Long numberOfOctets = fetchElement.getNumberOfOctets();
final String name = fetchElement.getResponseName();
@@ -268,7 +267,6 @@ public final class FetchResponseBuilder {
final Collection<String> names = fetchElement.getFieldNames();
final FetchResponse.BodyElement fullResult = bodyContent(messageResult, name, specifier, path, names);
return wrapIfPartialFetch(firstOctet, numberOfOctets, fullResult);
-
}
private FetchResponse.BodyElement bodyContent(MessageResult messageResult, String name, SectionType specifier, Optional<MimePath> path, Collection<String> names) throws MailboxException {
@@ -291,45 +289,38 @@ public final class FetchResponseBuilder {
}
private FetchResponse.BodyElement wrapIfPartialFetch(Long firstOctet, Long numberOfOctets, FetchResponse.BodyElement fullResult) {
- final FetchResponse.BodyElement result;
if (firstOctet == null) {
- result = fullResult;
+ return fullResult;
} else {
final long numberOfOctetsAsLong = Objects.requireNonNullElse(numberOfOctets, Long.MAX_VALUE);
final long firstOctetAsLong = firstOctet;
- result = new PartialFetchBodyElement(fullResult, firstOctetAsLong, numberOfOctetsAsLong);
-
-
+ return new PartialFetchBodyElement(fullResult, firstOctetAsLong, numberOfOctetsAsLong);
}
- return result;
}
private FetchResponse.BodyElement text(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
- final FetchResponse.BodyElement result;
- Content body;
+ Content body = Optional.ofNullable(getTextContent(messageResult, path))
+ .orElseGet(EmptyContent::new);
+ return new ContentBodyElement(name, body);
+ }
+
+ private Content getTextContent(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
if (!path.isPresent()) {
try {
- body = messageResult.getBody();
+ return messageResult.getBody();
} catch (IOException e) {
throw new MailboxException("Unable to get TEXT of body", e);
}
} else {
- body = messageResult.getBody(path.get());
- }
- if (body == null) {
- body = new EmptyContent();
+ return messageResult.getBody(path.get());
}
- result = new ContentBodyElement(name, body);
- return result;
}
private FetchResponse.BodyElement mimeHeaders(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
- final FetchResponse.BodyElement result;
final Iterator<Header> headers = getMimeHeaders(messageResult, path);
List<Header> lines = MessageResultUtils.getAll(headers);
- result = new MimeBodyElement(name, lines);
- return result;
+ return new MimeBodyElement(name, lines);
}
private HeaderBodyElement headerBodyElement(MessageResult messageResult, String name, List<Header> lines, Optional<MimePath> path) throws MailboxException {
@@ -397,13 +388,11 @@ public final class FetchResponseBuilder {
}
private Iterator<Header> getHeaders(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
- final Iterator<Header> headers;
if (!path.isPresent()) {
- headers = messageResult.getHeaders().headers();
+ return messageResult.getHeaders().headers();
} else {
- headers = messageResult.iterateHeaders(path.get());
+ return messageResult.iterateHeaders(path.get());
}
- return headers;
}
private Iterator<Header> getMimeHeaders(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
@@ -411,23 +400,21 @@ public final class FetchResponseBuilder {
}
private FetchResponse.BodyElement content(MessageResult messageResult, String name, Optional<MimePath> path) throws MailboxException {
- final FetchResponse.BodyElement result;
- Content full;
+ Content full = Optional.ofNullable(getContent(messageResult, path))
+ .orElseGet(EmptyContent::new);;
+ return new ContentBodyElement(name, full);
+ }
+
+ private Content getContent(MessageResult messageResult, Optional<MimePath> path) throws MailboxException {
if (!path.isPresent()) {
try {
- full = messageResult.getFullContent();
+ return messageResult.getFullContent();
} catch (IOException e) {
throw new MailboxException("Unable to get content", e);
}
} else {
- full = messageResult.getMimeBody(path.get());
+ return messageResult.getMimeBody(path.get());
}
-
- if (full == null) {
- full = new EmptyContent();
- }
- result = new ContentBodyElement(name, full);
- return result;
}
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 02/31: JAMES-3296 Add task to republish RabbitMQ
MailQueue from Cassandra
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7bc6a36334b4765d373b7610f26a1f1d044fbe28
Author: Rémi KOWALSKI <rk...@linagora.com>
AuthorDate: Tue Jul 7 17:32:23 2020 +0200
JAMES-3296 Add task to republish RabbitMQ MailQueue from Cassandra
---
pom.xml | 5 +
.../guice/cassandra-rabbitmq-guice/pom.xml | 4 +
.../james/CassandraRabbitMQJamesServerMain.java | 3 +-
server/container/guice/pom.xml | 6 +
.../webadmin-rabbitmq-mailqueue}/pom.xml | 25 ++-
.../server/RabbitMailQueueRoutesModule.java | 36 +++++
.../RabbitMailQueueTaskSerializationModule.java | 53 +++++++
server/container/guice/rabbitmq/pom.xml | 8 +
...dminServerTaskSerializationIntegrationTest.java | 23 ++-
server/protocols/webadmin/pom.xml | 1 +
.../protocols/webadmin/webadmin-mailqueue/pom.xml | 2 +-
.../pom.xml | 18 ++-
.../webadmin/routes/RabbitMQMailQueuesRoutes.java | 174 +++++++++++++++++++++
...ProcessedMailsTaskAdditionalInformationDTO.java | 90 +++++++++++
.../service/RepublishNotProcessedMailsTaskDTO.java | 85 ++++++++++
.../service/RepublishNotprocessedMailsTask.java | 107 +++++++++++++
.../routes/RabbitMQMailQueuesRoutesTest.java | 144 +++++++++++++++++
.../RepublishNotprocessedMailsTaskTest.java | 108 +++++++++++++
src/site/markdown/server/manage-webadmin.md | 35 +++++
19 files changed, 903 insertions(+), 24 deletions(-)
diff --git a/pom.xml b/pom.xml
index f2de229..0e7b225 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1877,6 +1877,11 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-webadmin-rabbitmq</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>james-server-webadmin-swagger</artifactId>
<version>${project.version}</version>
</dependency>
diff --git a/server/container/guice/cassandra-rabbitmq-guice/pom.xml b/server/container/guice/cassandra-rabbitmq-guice/pom.xml
index 1b02418..6edd29c 100644
--- a/server/container/guice/cassandra-rabbitmq-guice/pom.xml
+++ b/server/container/guice/cassandra-rabbitmq-guice/pom.xml
@@ -181,6 +181,10 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-webadmin-rabbitmq</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>testing-base</artifactId>
<scope>test</scope>
</dependency>
diff --git a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/CassandraRabbitMQJamesServerMain.java b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/CassandraRabbitMQJamesServerMain.java
index 2e40639..322acd3 100644
--- a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/CassandraRabbitMQJamesServerMain.java
+++ b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/CassandraRabbitMQJamesServerMain.java
@@ -29,6 +29,7 @@ import org.apache.james.modules.blobstore.BlobStoreModulesChooser;
import org.apache.james.modules.event.RabbitMQEventBusModule;
import org.apache.james.modules.rabbitmq.RabbitMQModule;
import org.apache.james.modules.server.JMXServerModule;
+import org.apache.james.modules.server.RabbitMailQueueRoutesModule;
import com.google.inject.Module;
import com.google.inject.util.Modules;
@@ -37,7 +38,7 @@ public class CassandraRabbitMQJamesServerMain implements JamesServerMain {
protected static final Module MODULES =
Modules
.override(Modules.combine(REQUIRE_TASK_MANAGER_MODULE, new DistributedTaskManagerModule()))
- .with(new RabbitMQModule(), new RabbitMQEventBusModule(), new DistributedTaskSerializationModule());
+ .with(new RabbitMQModule(), new RabbitMailQueueRoutesModule(), new RabbitMQEventBusModule(), new DistributedTaskSerializationModule());
public static void main(String[] args) throws Exception {
CassandraRabbitMQJamesConfiguration configuration = CassandraRabbitMQJamesConfiguration.builder()
diff --git a/server/container/guice/pom.xml b/server/container/guice/pom.xml
index bf8fe7f..cbae105 100644
--- a/server/container/guice/pom.xml
+++ b/server/container/guice/pom.xml
@@ -71,6 +71,7 @@
<module>protocols/webadmin-mailbox</module>
<module>protocols/webadmin-mailqueue</module>
<module>protocols/webadmin-mailrepository</module>
+ <module>protocols/webadmin-rabbitmq-mailqueue</module>
<module>protocols/webadmin-swagger</module>
<module>rabbitmq</module>
<module>testing</module>
@@ -200,6 +201,11 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-guice-webadmin-rabbitmq-mailqueue</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>james-server-guice-webadmin-mailrepository</artifactId>
<version>${project.version}</version>
</dependency>
diff --git a/server/container/guice/rabbitmq/pom.xml b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/pom.xml
similarity index 73%
copy from server/container/guice/rabbitmq/pom.xml
copy to server/container/guice/protocols/webadmin-rabbitmq-mailqueue/pom.xml
index b117e4f..9d625f4 100644
--- a/server/container/guice/rabbitmq/pom.xml
+++ b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/pom.xml
@@ -18,35 +18,29 @@
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
<modelVersion>4.0.0</modelVersion>
+
<parent>
- <groupId>org.apache.james</groupId>
<artifactId>james-server-guice</artifactId>
+ <groupId>org.apache.james</groupId>
<version>3.6.0-SNAPSHOT</version>
- <relativePath>../pom.xml</relativePath>
+ <relativePath>../../pom.xml</relativePath>
</parent>
- <artifactId>james-server-guice-rabbitmq</artifactId>
+ <artifactId>james-server-guice-webadmin-rabbitmq-mailqueue</artifactId>
- <name>Apache James :: Server :: Guice :: RabbitMQ</name>
- <description>Guice Module for RabbitMQ</description>
+ <name>Apache James :: Server :: Guice :: Webadmin :: RabbitMQ :: MailQueue</name>
+ <description>Webadmin rabbitMQ mailqueue modules for Guice implementation of James server</description>
<dependencies>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>apache-james-backends-rabbitmq</artifactId>
- </dependency>
- <dependency>
- <groupId>${james.groupId}</groupId>
- <artifactId>james-server-guice-configuration</artifactId>
+ <artifactId>james-server-webadmin-core</artifactId>
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>james-server-queue-api</artifactId>
- </dependency>
- <dependency>
- <groupId>${james.groupId}</groupId>
- <artifactId>james-server-queue-rabbitmq</artifactId>
+ <artifactId>james-server-webadmin-rabbitmq</artifactId>
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
@@ -58,5 +52,4 @@
<artifactId>guice</artifactId>
</dependency>
</dependencies>
-
</project>
diff --git a/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueRoutesModule.java b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueRoutesModule.java
new file mode 100644
index 0000000..dbffe76
--- /dev/null
+++ b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueRoutesModule.java
@@ -0,0 +1,36 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.modules.server;
+
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.routes.RabbitMQMailQueuesRoutes;
+
+import com.google.inject.AbstractModule;
+import com.google.inject.multibindings.Multibinder;
+
+public class RabbitMailQueueRoutesModule extends AbstractModule {
+ @Override
+ protected void configure() {
+ install(new RabbitMailQueueTaskSerializationModule());
+
+ Multibinder<Routes> routesMultibinder = Multibinder.newSetBinder(binder(), Routes.class);
+ routesMultibinder.addBinding().to(RabbitMQMailQueuesRoutes.class);
+ }
+}
diff --git a/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueTaskSerializationModule.java b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueTaskSerializationModule.java
new file mode 100644
index 0000000..61d7048
--- /dev/null
+++ b/server/container/guice/protocols/webadmin-rabbitmq-mailqueue/src/main/java/org/apache/james/modules/server/RabbitMailQueueTaskSerializationModule.java
@@ -0,0 +1,53 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.modules.server;
+
+import org.apache.james.queue.api.MailQueueFactory;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTO;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule;
+import org.apache.james.server.task.json.dto.TaskDTO;
+import org.apache.james.server.task.json.dto.TaskDTOModule;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.webadmin.dto.DTOModuleInjections;
+import org.apache.james.webadmin.service.RepublishNotProcessedMailsTaskAdditionalInformationDTO;
+import org.apache.james.webadmin.service.RepublishNotProcessedMailsTaskDTO;
+
+import com.google.inject.AbstractModule;
+import com.google.inject.multibindings.ProvidesIntoSet;
+import com.google.inject.name.Named;
+
+public class RabbitMailQueueTaskSerializationModule extends AbstractModule {
+ @ProvidesIntoSet
+ public TaskDTOModule<? extends Task, ? extends TaskDTO> republishNotProcessedMailsTask(MailQueueFactory<RabbitMQMailQueue> mailQueueFactory) {
+ return RepublishNotProcessedMailsTaskDTO.module(mailQueueFactory);
+ }
+
+ @ProvidesIntoSet
+ public AdditionalInformationDTOModule<? extends TaskExecutionDetails.AdditionalInformation, ? extends AdditionalInformationDTO> republishNotProcessedMailsAdditionalInformation() {
+ return RepublishNotProcessedMailsTaskAdditionalInformationDTO.module();
+ }
+
+ @Named(DTOModuleInjections.WEBADMIN_DTO)
+ @ProvidesIntoSet
+ public AdditionalInformationDTOModule<? extends TaskExecutionDetails.AdditionalInformation, ? extends AdditionalInformationDTO> republishNotProcessedMailsAdditionalInformationWebAdmin() {
+ return RepublishNotProcessedMailsTaskAdditionalInformationDTO.module();
+ }
+}
diff --git a/server/container/guice/rabbitmq/pom.xml b/server/container/guice/rabbitmq/pom.xml
index b117e4f..5c4b0cc 100644
--- a/server/container/guice/rabbitmq/pom.xml
+++ b/server/container/guice/rabbitmq/pom.xml
@@ -42,6 +42,14 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-guice-webadmin-rabbitmq-mailqueue</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-webadmin-rabbitmq</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>james-server-queue-api</artifactId>
</dependency>
<dependency>
diff --git a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerTaskSerializationIntegrationTest.java b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerTaskSerializationIntegrationTest.java
index f0775df..3d5e3ed 100644
--- a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerTaskSerializationIntegrationTest.java
+++ b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerTaskSerializationIntegrationTest.java
@@ -797,4 +797,25 @@ class RabbitMQWebAdminServerTaskSerializationIntegrationTest {
.mailboxPath(MailboxPath.forUser(Username.of(USERNAME), "Important-mailbox"))
.build();
}
-}
\ No newline at end of file
+
+ @Test
+ void republishNotProcessedMailsOnSpoolShouldComplete() {
+ String taskId = with()
+ .basePath("/mailQueues/spool")
+ .queryParam("action", "RepublishNotProcessedMails")
+ .queryParam("olderThan", "2d")
+ .post()
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await")
+ .then()
+ .body("status", is("completed"))
+ .body("taskId", is(taskId))
+ .body("type", is("republish-not-processed-mails"))
+ .body("additionalInformation.nbRequeuedMails", is(0));
+ }
+}
diff --git a/server/protocols/webadmin/pom.xml b/server/protocols/webadmin/pom.xml
index 4600e0a..db7f387 100644
--- a/server/protocols/webadmin/pom.xml
+++ b/server/protocols/webadmin/pom.xml
@@ -42,6 +42,7 @@
<module>webadmin-mailbox-deleted-message-vault</module>
<module>webadmin-mailqueue</module>
<module>webadmin-mailrepository</module>
+ <module>webadmin-rabbitmq</module>
<module>webadmin-swagger</module>
</modules>
diff --git a/server/protocols/webadmin/webadmin-mailqueue/pom.xml b/server/protocols/webadmin/webadmin-mailqueue/pom.xml
index f119ef2..a35a339 100644
--- a/server/protocols/webadmin/webadmin-mailqueue/pom.xml
+++ b/server/protocols/webadmin/webadmin-mailqueue/pom.xml
@@ -129,7 +129,7 @@
<version>v1</version>
</info>
<swaggerDirectory>${project.build.directory}</swaggerDirectory>
- <swaggerFileName>webadmin-mailbox</swaggerFileName>
+ <swaggerFileName>webadmin-mailqueue</swaggerFileName>
</apiSource>
</apiSources>
</configuration>
diff --git a/server/protocols/webadmin/webadmin-mailqueue/pom.xml b/server/protocols/webadmin/webadmin-rabbitmq/pom.xml
similarity index 88%
copy from server/protocols/webadmin/webadmin-mailqueue/pom.xml
copy to server/protocols/webadmin/webadmin-rabbitmq/pom.xml
index f119ef2..862deff 100644
--- a/server/protocols/webadmin/webadmin-mailqueue/pom.xml
+++ b/server/protocols/webadmin/webadmin-rabbitmq/pom.xml
@@ -27,10 +27,10 @@
<relativePath>../../../pom.xml</relativePath>
</parent>
- <artifactId>james-server-webadmin-mailqueue</artifactId>
+ <artifactId>james-server-webadmin-rabbitmq</artifactId>
<packaging>jar</packaging>
- <name>Apache James :: Server :: Web Admin :: MailQueue</name>
+ <name>Apache James :: Server :: Web Admin :: RabbitMQ</name>
<dependencies>
<dependency>
@@ -56,7 +56,11 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>james-server-queue-memory</artifactId>
+ <artifactId>james-server-queue-rabbitmq</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-testing</artifactId>
<scope>test</scope>
</dependency>
<dependency>
@@ -65,7 +69,7 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>james-server-task-memory</artifactId>
+ <artifactId>james-server-task-distributed</artifactId>
<scope>test</scope>
</dependency>
<dependency>
@@ -80,6 +84,10 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-webadmin-mailqueue</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>metrics-tests</artifactId>
<scope>test</scope>
</dependency>
@@ -129,7 +137,7 @@
<version>v1</version>
</info>
<swaggerDirectory>${project.build.directory}</swaggerDirectory>
- <swaggerFileName>webadmin-mailbox</swaggerFileName>
+ <swaggerFileName>webadmin-rabbitmq</swaggerFileName>
</apiSource>
</apiSources>
</configuration>
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutes.java b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutes.java
new file mode 100644
index 0000000..c366df6
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutes.java
@@ -0,0 +1,174 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.webadmin.routes;
+
+import static org.apache.james.webadmin.Constants.SEPARATOR;
+import static org.apache.james.webadmin.routes.MailQueueRoutes.BASE_URL;
+import static org.apache.james.webadmin.routes.MailQueueRoutes.MAIL_QUEUE_NAME;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.function.Predicate;
+
+import javax.inject.Inject;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+
+import org.apache.james.queue.api.MailQueueFactory;
+import org.apache.james.queue.api.MailQueueName;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskManager;
+import org.apache.james.util.DurationParser;
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.service.RepublishNotprocessedMailsTask;
+import org.apache.james.webadmin.tasks.TaskFromRequest;
+import org.apache.james.webadmin.tasks.TaskFromRequestRegistry;
+import org.apache.james.webadmin.tasks.TaskRegistrationKey;
+import org.apache.james.webadmin.utils.ErrorResponder;
+import org.apache.james.webadmin.utils.JsonTransformer;
+import org.eclipse.jetty.http.HttpStatus;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiImplicitParam;
+import io.swagger.annotations.ApiImplicitParams;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import spark.Request;
+import spark.Service;
+
+@Api(tags = "MailQueues")
+@Path(BASE_URL)
+@Produces("application/json")
+public class RabbitMQMailQueuesRoutes implements Routes {
+
+ private static final TaskRegistrationKey REPUBLISH_NOT_PROCESSED_MAILS_REGISTRATION_KEY = TaskRegistrationKey.of("RepublishNotProcessedMails");
+
+ private final MailQueueFactory<RabbitMQMailQueue> mailQueueFactory;
+ private final JsonTransformer jsonTransformer;
+ private final TaskManager taskManager;
+ private final Clock clock;
+
+ @Inject
+ @SuppressWarnings("unchecked")
+ @VisibleForTesting
+ RabbitMQMailQueuesRoutes(MailQueueFactory<RabbitMQMailQueue> mailQueueFactory,
+ Clock clock, JsonTransformer jsonTransformer, TaskManager taskManager) {
+ this.mailQueueFactory = mailQueueFactory;
+ this.clock = clock;
+ this.jsonTransformer = jsonTransformer;
+ this.taskManager = taskManager;
+ }
+
+ @Override
+ public String getBasePath() {
+ return BASE_URL;
+ }
+
+ @Override
+ public void define(Service service) {
+ republishNotProcessedMails(service);
+ }
+
+
+ @POST
+ @Path("/{mailQueueName}")
+ @ApiImplicitParams({
+ @ApiImplicitParam(required = true, dataType = "string", name = "mailQueueName", paramType = "path"),
+ @ApiImplicitParam(
+ required = true,
+ dataType = "String",
+ name = "action",
+ paramType = "query",
+ example = "?action=RepublishNotProcessedMails",
+ value = "Specify the action to perform on a RabbitMQ mail queue."),
+ @ApiImplicitParam(
+ required = true,
+ dataType = "String",
+ name = "olderThan",
+ paramType = "query",
+ example = "?olderThan=1w",
+ value = "Specify the messages minimum age to republish")
+ })
+ @ApiOperation(
+ value = "republish the not processed mails of the RabbitMQ MailQueue using the cassandra mail queue view"
+ )
+ @ApiResponses(value = {
+ @ApiResponse(code = HttpStatus.CREATED_201, message = "OK, the task for rebuilding the queue is created"),
+ @ApiResponse(code = HttpStatus.BAD_REQUEST_400, message = "Invalid request for rebuilding the mail queue."),
+ @ApiResponse(code = HttpStatus.INTERNAL_SERVER_ERROR_500, message = "Internal server error - Something went bad on the server side.")
+ })
+ public void republishNotProcessedMails(Service service) {
+ TaskFromRequest taskFromRequest = this::republishNotProcessedMails;
+ service.post(BASE_URL + SEPARATOR + MAIL_QUEUE_NAME,
+ TaskFromRequestRegistry.builder()
+ .register(REPUBLISH_NOT_PROCESSED_MAILS_REGISTRATION_KEY, this::republishNotProcessedMails)
+ .buildAsRoute(taskManager),
+ jsonTransformer);
+ }
+
+ private Task republishNotProcessedMails(Request request) {
+ RabbitMQMailQueue mailQueue = getMailQueue(MailQueueName.of(request.params(MAIL_QUEUE_NAME)));
+ return new RepublishNotprocessedMailsTask(mailQueue, getOlderThan(request));
+ }
+
+
+ private RabbitMQMailQueue getMailQueue(MailQueueName mailQueueName) {
+ return mailQueueFactory.getQueue(mailQueueName)
+ .orElseThrow(
+ () -> ErrorResponder.builder()
+ .message("%s can not be found", mailQueueName)
+ .statusCode(HttpStatus.NOT_FOUND_404)
+ .type(ErrorResponder.ErrorType.NOT_FOUND)
+ .haltError());
+ }
+
+ private Instant getOlderThan(Request req) {
+ try {
+ Duration olderThan = Optional.ofNullable(req.queryParams("olderThan"))
+ .filter(Predicate.not(String::isEmpty))
+ .map(rawString -> DurationParser.parse(rawString, ChronoUnit.DAYS))
+ .orElseThrow();
+
+ return clock.instant().minus(olderThan);
+ } catch (NoSuchElementException e) {
+ throw ErrorResponder.builder()
+ .message("Missing olderThan")
+ .statusCode(HttpStatus.BAD_REQUEST_400)
+ .type(ErrorResponder.ErrorType.INVALID_ARGUMENT)
+ .haltError();
+ } catch (Exception e) {
+ throw ErrorResponder.builder()
+ .statusCode(HttpStatus.BAD_REQUEST_400)
+ .cause(e)
+ .type(ErrorResponder.ErrorType.INVALID_ARGUMENT)
+ .message("Invalid olderThan")
+ .haltError();
+ }
+ }
+}
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskAdditionalInformationDTO.java b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskAdditionalInformationDTO.java
new file mode 100644
index 0000000..3621530
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskAdditionalInformationDTO.java
@@ -0,0 +1,90 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.webadmin.service;
+
+import java.time.Instant;
+
+import org.apache.james.json.DTOModule;
+import org.apache.james.queue.api.MailQueueName;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTO;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class RepublishNotProcessedMailsTaskAdditionalInformationDTO implements AdditionalInformationDTO {
+
+ public static AdditionalInformationDTOModule<RepublishNotprocessedMailsTask.AdditionalInformation, RepublishNotProcessedMailsTaskAdditionalInformationDTO> module() {
+ return DTOModule.forDomainObject(RepublishNotprocessedMailsTask.AdditionalInformation.class)
+ .convertToDTO(RepublishNotProcessedMailsTaskAdditionalInformationDTO.class)
+ .toDomainObjectConverter(dto -> new RepublishNotprocessedMailsTask.AdditionalInformation(
+ MailQueueName.of(dto.mailQueue),
+ dto.olderThan,
+ dto.nbRequeuedMails,
+ dto.timestamp))
+ .toDTOConverter((details, type) -> new RepublishNotProcessedMailsTaskAdditionalInformationDTO(
+ type,
+ details.getMailQueue().asString(),
+ details.getOlderThan(),
+ details.getNbRequeuedMails(),
+ details.timestamp()))
+ .typeName(RepublishNotprocessedMailsTask.TYPE.asString())
+ .withFactory(AdditionalInformationDTOModule::new);
+ }
+
+ private final String type;
+ private final String mailQueue;
+
+ private final long nbRequeuedMails;
+ private final Instant olderThan;
+ private final Instant timestamp;
+
+ public RepublishNotProcessedMailsTaskAdditionalInformationDTO(@JsonProperty("type") String type,
+ @JsonProperty("mailQueue") String mailQueue,
+ @JsonProperty("olderThan") Instant olderThan,
+ @JsonProperty("nbRequeuedMails") long nbRequeuedMails,
+ @JsonProperty("timestamp") Instant timestamp) {
+ this.type = type;
+ this.mailQueue = mailQueue;
+ this.olderThan = olderThan;
+ this.nbRequeuedMails = nbRequeuedMails;
+ this.timestamp = timestamp;
+ }
+
+ public String getMailQueue() {
+ return mailQueue;
+ }
+
+ public long getNbRequeuedMails() {
+ return nbRequeuedMails;
+ }
+
+ public Instant getOlderThan() {
+ return olderThan;
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+
+ @Override
+ public Instant getTimestamp() {
+ return timestamp;
+ }
+}
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskDTO.java b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskDTO.java
new file mode 100644
index 0000000..5502b35
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotProcessedMailsTaskDTO.java
@@ -0,0 +1,85 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+package org.apache.james.webadmin.service;
+
+import java.time.Instant;
+
+import org.apache.james.json.DTOModule;
+import org.apache.james.queue.api.MailQueueFactory;
+import org.apache.james.queue.api.MailQueueName;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.server.task.json.dto.TaskDTO;
+import org.apache.james.server.task.json.dto.TaskDTOModule;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class RepublishNotProcessedMailsTaskDTO implements TaskDTO {
+
+ public static class UnknownMailQueueException extends RuntimeException {
+ public UnknownMailQueueException(MailQueueName mailQueueName) {
+ super("Unknown mail queue " + mailQueueName.asString());
+ }
+ }
+
+ public static TaskDTOModule<RepublishNotprocessedMailsTask, RepublishNotProcessedMailsTaskDTO> module(MailQueueFactory<RabbitMQMailQueue> mailQueueFactory) {
+ return DTOModule
+ .forDomainObject(RepublishNotprocessedMailsTask.class)
+ .convertToDTO(RepublishNotProcessedMailsTaskDTO.class)
+ .toDomainObjectConverter(dto -> dto.fromDTO(mailQueueFactory))
+ .toDTOConverter(RepublishNotProcessedMailsTaskDTO::toDTO)
+ .typeName(RepublishNotprocessedMailsTask.TYPE.asString())
+ .withFactory(TaskDTOModule::new);
+ }
+
+ public static RepublishNotProcessedMailsTaskDTO toDTO(RepublishNotprocessedMailsTask domainObject, String typeName) {
+ return new RepublishNotProcessedMailsTaskDTO(typeName, domainObject.getMailQueue().asString(), domainObject.getOlderThan());
+ }
+
+ private final String type;
+ private final String mailQueue;
+ private final Instant olderThan;
+
+ public RepublishNotProcessedMailsTaskDTO(@JsonProperty("type") String type, @JsonProperty("mailQueue") String mailQueue, @JsonProperty("olderThan") Instant olderThan) {
+ this.type = type;
+ this.mailQueue = mailQueue;
+ this.olderThan = olderThan;
+ }
+
+ public RepublishNotprocessedMailsTask fromDTO(MailQueueFactory<RabbitMQMailQueue> mailQueueFactory) {
+ MailQueueName requestedMailQueueName = MailQueueName.of(mailQueue);
+ RabbitMQMailQueue queue = mailQueueFactory
+ .getQueue(requestedMailQueueName)
+ .orElseThrow(() -> new UnknownMailQueueException(requestedMailQueueName));
+
+ return new RepublishNotprocessedMailsTask(queue, olderThan);
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+
+ public Instant getOlderThan() {
+ return olderThan;
+ }
+
+ public String getMailQueue() {
+ return mailQueue;
+ }
+}
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTask.java b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTask.java
new file mode 100644
index 0000000..0428335
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/main/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTask.java
@@ -0,0 +1,107 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.webadmin.service;
+
+import java.time.Clock;
+import java.time.Instant;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.james.queue.api.MailQueueName;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+
+
+public class RepublishNotprocessedMailsTask implements Task {
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+
+ private final Instant timestamp;
+ private final long nbRequeuedMails;
+ private final MailQueueName mailQueue;
+ private final Instant olderThan;
+
+ public AdditionalInformation(MailQueueName mailQueue, Instant olderThan, long nbRequeuedMails, Instant timestamp) {
+ this.mailQueue = mailQueue;
+ this.olderThan = olderThan;
+ this.timestamp = timestamp;
+ this.nbRequeuedMails = nbRequeuedMails;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+
+ public Instant getOlderThan() {
+ return olderThan;
+ }
+
+ public MailQueueName getMailQueue() {
+ return mailQueue;
+ }
+
+ public long getNbRequeuedMails() {
+ return nbRequeuedMails;
+ }
+ }
+
+ public static final TaskType TYPE = TaskType.of("republish-not-processed-mails");
+
+ private final Instant olderThan;
+ private final RabbitMQMailQueue mailQueue;
+ private final AtomicInteger nbRequeuedMails;
+
+ public RepublishNotprocessedMailsTask(RabbitMQMailQueue mailQueue, Instant olderThan) {
+ this.olderThan = olderThan;
+ this.mailQueue = mailQueue;
+ this.nbRequeuedMails = new AtomicInteger(0);
+ }
+
+ @Override
+ public Result run() {
+ mailQueue.republishNotProcessedMails(olderThan)
+ .doOnNext(mailName -> nbRequeuedMails.getAndIncrement())
+ .then()
+ .block();
+
+ return Result.COMPLETED;
+ }
+
+ @Override
+ public TaskType type() {
+ return TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.of(new AdditionalInformation(mailQueue.getName(), olderThan, nbRequeuedMails.get(), Clock.systemUTC().instant()));
+ }
+
+ public Instant getOlderThan() {
+ return olderThan;
+ }
+
+ public MailQueueName getMailQueue() {
+ return mailQueue.getName();
+ }
+}
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutesTest.java b/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutesTest.java
new file mode 100644
index 0000000..4daaa08
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/routes/RabbitMQMailQueuesRoutesTest.java
@@ -0,0 +1,144 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.webadmin.routes;
+
+import static io.restassured.RestAssured.given;
+import static io.restassured.config.EncoderConfig.encoderConfig;
+import static io.restassured.config.RestAssuredConfig.newConfig;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.hasSize;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Clock;
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.util.Optional;
+
+import org.apache.james.json.DTOConverter;
+import org.apache.james.queue.api.MailQueueFactory;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueueFactory;
+import org.apache.james.task.Hostname;
+import org.apache.james.task.MemoryTaskManager;
+import org.apache.james.task.TaskManager;
+import org.apache.james.utils.UpdatableTickingClock;
+import org.apache.james.webadmin.WebAdminServer;
+import org.apache.james.webadmin.WebAdminUtils;
+import org.apache.james.webadmin.utils.JsonTransformer;
+import org.eclipse.jetty.http.HttpStatus;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import io.restassured.RestAssured;
+import io.restassured.builder.RequestSpecBuilder;
+import io.restassured.http.ContentType;
+import io.restassured.specification.RequestSpecification;
+
+class RabbitMQMailQueuesRoutesTest {
+ final static ZonedDateTime DATE = ZonedDateTime.parse("2015-10-30T14:12:00Z");
+
+ WebAdminServer webAdminServer;
+ MailQueueFactory mailQueueFactory;
+ Clock clock;
+
+ WebAdminServer createServer(MailQueueFactory mailQueueFactory) {
+ TaskManager taskManager = new MemoryTaskManager(new Hostname("foo"));
+ JsonTransformer jsonTransformer = new JsonTransformer();
+ clock = UpdatableTickingClock.fixed(DATE.toInstant(), ZoneOffset.UTC);
+ return WebAdminUtils.createWebAdminServer(
+ new RabbitMQMailQueuesRoutes(mailQueueFactory, clock, jsonTransformer, taskManager),
+ new TasksRoutes(taskManager, jsonTransformer, DTOConverter.of()))
+ .start();
+ }
+
+ RequestSpecification buildRequestSpecification(WebAdminServer server) {
+ return new RequestSpecBuilder()
+ .setContentType(ContentType.JSON)
+ .setAccept(ContentType.JSON)
+ .setBasePath("/")
+ .setPort(server.getPort().getValue())
+ .setConfig(newConfig().encoderConfig(encoderConfig().defaultContentCharset(StandardCharsets.UTF_8)))
+ .build();
+ }
+
+ @BeforeEach
+ void setUp() {
+ mailQueueFactory = mock(RabbitMQMailQueueFactory.class);
+ webAdminServer = createServer(mailQueueFactory);
+ RestAssured.requestSpecification = buildRequestSpecification(webAdminServer);
+ RestAssured.enableLoggingOfRequestAndResponseIfValidationFails();
+ }
+
+ @AfterEach
+ void tearDown() {
+ webAdminServer.destroy();
+ }
+
+ @Test
+ void triggeringARepublishNotProcessedMailsShouldCreateATask() {
+ when(mailQueueFactory.getQueue(any())).thenReturn(Optional.of(mock(RabbitMQMailQueue.class)));
+ given()
+ .queryParam("action", "RepublishNotProcessedMails")
+ .queryParam("olderThan", "1d")
+ .when()
+ .post(MailQueueRoutes.BASE_URL + "/spooler")
+ .then()
+ .statusCode(HttpStatus.CREATED_201);
+
+ given()
+ .when()
+ .get("/tasks")
+ .then()
+ .statusCode(HttpStatus.OK_200)
+ .body("", hasSize(1));
+ }
+
+ @Test
+ void triggeringARepublishNotProcessedMailsWhenTheQueueHasNotBeenInitializedShouldFail() {
+ when(mailQueueFactory.getQueue(any())).thenReturn(Optional.empty());
+ given()
+ .queryParam("action", "RepublishNotProcessedMails")
+ .queryParam("olderThan", "1d")
+ .when()
+ .post(MailQueueRoutes.BASE_URL + "/spooler")
+ .then()
+ .statusCode(HttpStatus.NOT_FOUND_404)
+ .body("message", containsString("MailQueueName{value=spooler} can not be found"));
+ }
+
+ @Test
+ void triggeringARepublishNotProcessedMailsWithAnInvalidOlderThanShouldFail() {
+ when(mailQueueFactory.getQueue(any())).thenReturn(Optional.of(mock(RabbitMQMailQueue.class)));
+ given()
+ .queryParam("action", "RepublishNotProcessedMails")
+ .queryParam("olderThan", "invalidValue")
+ .when()
+ .post(MailQueueRoutes.BASE_URL + "/spooler")
+ .then()
+ .statusCode(HttpStatus.BAD_REQUEST_400)
+ .body("message", containsString("Invalid olderThan"))
+ .body("details", containsString("Supplied value do not follow the unit format (number optionally suffixed with a string representing the unit"));
+ }
+
+}
diff --git a/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTaskTest.java b/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTaskTest.java
new file mode 100644
index 0000000..ac35da1
--- /dev/null
+++ b/server/protocols/webadmin/webadmin-rabbitmq/src/test/java/org/apache/james/webadmin/service/RepublishNotprocessedMailsTaskTest.java
@@ -0,0 +1,108 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ * ***************************************************************/
+
+package org.apache.james.webadmin.service;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.time.Instant;
+import java.util.Optional;
+
+import org.apache.james.JsonSerializationVerifier;
+import org.apache.james.json.JsonGenericSerializer;
+import org.apache.james.queue.api.MailQueueName;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueue;
+import org.apache.james.queue.rabbitmq.RabbitMQMailQueueFactory;
+import org.junit.jupiter.api.Test;
+
+class RepublishNotprocessedMailsTaskTest {
+ private static final Instant OLDER_THAN = Instant.parse("2018-11-13T12:00:55Z");
+ private static final Instant NOW = Instant.now();
+ private static final long NB_REQUEUED_MAILS = 12;
+ private static final String SERIALIZED = "{\"type\": \"republish-not-processed-mails\",\"mailQueue\":\"anyQueue\", \"olderThan\": \"" + OLDER_THAN + "\"}";
+ private static final String SERIALIZED_TASK_ADDITIONAL_INFORMATION = "{\"type\": \"republish-not-processed-mails\",\"mailQueue\":\"anyQueue\", \"olderThan\": \"" + OLDER_THAN + "\" ,\"nbRequeuedMails\":12,\"timestamp\":\"" + NOW.toString() + "\"}";
+ private static final MailQueueName QUEUE_NAME = MailQueueName.of("anyQueue");
+
+ @Test
+ void taskShouldBeSerializable() throws Exception {
+ RabbitMQMailQueueFactory mockedQueueFactory = mock(RabbitMQMailQueueFactory.class);
+ RabbitMQMailQueue mockedQueue = mock(RabbitMQMailQueue.class);
+
+ when(mockedQueue.getName()).thenReturn(QUEUE_NAME);
+ when(mockedQueueFactory.getQueue(QUEUE_NAME)).thenReturn(Optional.of(mockedQueue));
+
+ RepublishNotprocessedMailsTask task = new RepublishNotprocessedMailsTask(mockedQueue, OLDER_THAN);
+
+ JsonSerializationVerifier.dtoModule(RepublishNotProcessedMailsTaskDTO.module(mockedQueueFactory))
+ .bean(task)
+ .json(SERIALIZED)
+ .verify();
+ }
+
+ @Test
+ void taskShouldBeDeserializable() throws Exception {
+ RabbitMQMailQueueFactory mockedQueueFactory = mock(RabbitMQMailQueueFactory.class);
+ RabbitMQMailQueue mockedQueue = mock(RabbitMQMailQueue.class);
+
+ when(mockedQueue.getName()).thenReturn(QUEUE_NAME);
+ when(mockedQueueFactory.getQueue(QUEUE_NAME)).thenReturn(Optional.of(mockedQueue));
+
+ RepublishNotprocessedMailsTask task = new RepublishNotprocessedMailsTask(mockedQueue, OLDER_THAN);
+ JsonSerializationVerifier.dtoModule(RepublishNotProcessedMailsTaskDTO.module(mockedQueueFactory))
+ .bean(task)
+ .json(SERIALIZED)
+ .verify();
+ }
+
+ @Test
+ void taskDeserializationFromUnknownQueueNameShouldThrow() {
+ RabbitMQMailQueueFactory mockedQueueFactory = mock(RabbitMQMailQueueFactory.class);
+ RabbitMQMailQueue mockedQueue = mock(RabbitMQMailQueue.class);
+
+ when(mockedQueue.getName()).thenReturn(QUEUE_NAME);
+ when(mockedQueueFactory.getQueue(QUEUE_NAME)).thenReturn(Optional.empty());
+
+ RepublishNotprocessedMailsTask task = new RepublishNotprocessedMailsTask(mockedQueue, OLDER_THAN);
+ assertThatThrownBy(() -> JsonSerializationVerifier.dtoModule(RepublishNotProcessedMailsTaskDTO.module(mockedQueueFactory))
+ .bean(task)
+ .json(SERIALIZED)
+ .verify())
+ .isInstanceOf(RepublishNotProcessedMailsTaskDTO.UnknownMailQueueException.class);
+ }
+
+ @Test
+ void additionalInformationShouldBeSerializable() throws Exception {
+ RepublishNotprocessedMailsTask.AdditionalInformation details = new RepublishNotprocessedMailsTask.AdditionalInformation(QUEUE_NAME, OLDER_THAN, NB_REQUEUED_MAILS, NOW);
+ JsonSerializationVerifier.dtoModule(RepublishNotProcessedMailsTaskAdditionalInformationDTO.module())
+ .bean(details)
+ .json(SERIALIZED_TASK_ADDITIONAL_INFORMATION)
+ .verify();
+ }
+
+ @Test
+ void additionalInformationShouldBeDeserializable() throws Exception {
+ RepublishNotprocessedMailsTask.AdditionalInformation details = new RepublishNotprocessedMailsTask.AdditionalInformation(QUEUE_NAME, OLDER_THAN, NB_REQUEUED_MAILS, NOW);
+ RepublishNotprocessedMailsTask.AdditionalInformation deserialized = JsonGenericSerializer.forModules(RepublishNotProcessedMailsTaskAdditionalInformationDTO.module())
+ .withoutNestedType()
+ .deserialize(SERIALIZED_TASK_ADDITIONAL_INFORMATION);
+ assertThat(deserialized).isEqualToComparingFieldByField(details);
+ }
+}
diff --git a/src/site/markdown/server/manage-webadmin.md b/src/site/markdown/server/manage-webadmin.md
index 707417d..ff18829 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -2900,6 +2900,7 @@ The scheduled task will have the following type `reprocessing-one` and the follo
- [Deleting mails from a mail queue](#Deleting_mails_from_a_mail_queue)
- [Clearing a mail queue](#Clearing_a_mail_queue)
- [Flushing mails from a mail queue](#Flushing_mails_from_a_mail_queue)
+ - [RabbitMQ republishing a mail queue from cassandra](#RabbitMQ_republishing_a_mail_queue_from_cassandra)
### Listing mail queues
@@ -3048,6 +3049,40 @@ Response codes:
- 204: Success (No content)
- 400: Invalid request
- 404: The mail queue does not exist
+
+### RabbitMQ republishing a mail queue from cassandra
+
+```
+curl -XPOST 'http://ip:port/mailQueues/{mailQueueName}?action=RepublishNotProcessedMails&olderThan=1d'
+```
+
+This method is specific to the distributed flavor of James, which relies on Cassandra and RabbitMQ for implementing a mail queue.
+In case of a RabbitMQ crash resulting in a loss of messages, this task can be launched to repopulate the
+`mailQueueName` queue in RabbitMQ using the information stored in Cassandra.
+
+The `olderThan` parameter is mandatory. It filters the mails to be restored, by taking into account only
+the mails older than the given value.
+The expected value should be expressed in the following format: `Nunit`.
+`N` should be strictly positive.
+`unit` could be either in the short form (`h`, `d`, `w`, etc.), or in the long form (`day`, `week`, `month`, etc.).
+
+Examples:
+
+ - `5h`
+ - `7d`
+ - `1y`
+
+Response codes:
+
+ - 201: Task created
+ - 400: Invalid request
+
+ The response body contains the id of the republishing task.
+ ```
+ {
+ "taskId": "a650a66a-5984-431e-bdad-f1baad885856"
+ }
+ ```
## Administrating DLP Configuration
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 12/31: [REFACTORING] Remove unused
FetchResponse.Address empty array
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit b531212738943d1e7b98e7b76f24b806184bce29
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 16 11:28:10 2020 +0700
[REFACTORING] Remove unused FetchResponse.Address empty array
---
.../java/org/apache/james/imap/message/response/FetchResponse.java | 3 ---
1 file changed, 3 deletions(-)
diff --git a/protocols/imap/src/main/java/org/apache/james/imap/message/response/FetchResponse.java b/protocols/imap/src/main/java/org/apache/james/imap/message/response/FetchResponse.java
index 1b5133f..67697e1 100644
--- a/protocols/imap/src/main/java/org/apache/james/imap/message/response/FetchResponse.java
+++ b/protocols/imap/src/main/java/org/apache/james/imap/message/response/FetchResponse.java
@@ -377,9 +377,6 @@ public final class FetchResponse implements ImapResponseMessage {
*/
interface Address {
- /** Empty array */
- Address[] EMPTY = {};
-
/**
* Gets the personal name.
*
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 07/31: JAMES−2290 Fix unstable test: DiscreteDistributionTest.partitionShouldSupportDuplicatedDistributionEntry
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit add3e9a24abdfef3e25df1f615ca3d5e09cd0f44
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Jul 15 13:56:10 2020 +0700
JAMES−2290 Fix unstable test: DiscreteDistributionTest.partitionShouldSupportDuplicatedDistributionEntry
We encountered the following failure:
```
[ERROR] DiscreteDistributionTest.partitionShouldSupportDuplicatedDistributionEntry:112
Expecting:
<668532L>
to be close to:
<662936L>
by less than <5000L> but difference was <5596L>.
(a difference of exactly <5000L> being considered valid)
```
A difference of 10.000 seems reasonable too and more unlikely to break.
---
.../main/java/org/apache/james/utils/DiscreteDistribution.java | 3 ++-
.../java/org/apache/james/utils/DiscreteDistributionTest.java | 10 +++++++---
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/server/testing/src/main/java/org/apache/james/utils/DiscreteDistribution.java b/server/testing/src/main/java/org/apache/james/utils/DiscreteDistribution.java
index eaeaaf5..f3fbdbc 100644
--- a/server/testing/src/main/java/org/apache/james/utils/DiscreteDistribution.java
+++ b/server/testing/src/main/java/org/apache/james/utils/DiscreteDistribution.java
@@ -23,6 +23,7 @@ import java.util.List;
import java.util.stream.Stream;
import org.apache.commons.math3.distribution.EnumeratedDistribution;
+import org.apache.commons.math3.random.MersenneTwister;
import org.apache.commons.math3.util.Pair;
import com.github.steveash.guavate.Guavate;
@@ -66,7 +67,7 @@ public class DiscreteDistribution<T> {
private final EnumeratedDistribution<T> enumeratedDistribution;
private DiscreteDistribution(List<DistributionEntry<T>> distribution) {
- enumeratedDistribution = new EnumeratedDistribution<>(distribution.stream()
+ enumeratedDistribution = new EnumeratedDistribution<>(new MersenneTwister(), distribution.stream()
.map(DistributionEntry::toPair)
.collect(Guavate.toImmutableList()));
}
diff --git a/server/testing/src/test/java/org/apache/james/utils/DiscreteDistributionTest.java b/server/testing/src/test/java/org/apache/james/utils/DiscreteDistributionTest.java
index 4aaf150..864fbd7 100644
--- a/server/testing/src/test/java/org/apache/james/utils/DiscreteDistributionTest.java
+++ b/server/testing/src/test/java/org/apache/james/utils/DiscreteDistributionTest.java
@@ -33,6 +33,7 @@ import org.junit.jupiter.api.Test;
import com.google.common.collect.ImmutableList;
class DiscreteDistributionTest {
+ public static final Offset<Long> OFFSET = Offset.offset(10_000L);
@Test
void createShouldNotSupportNegativeDistribution() {
@@ -84,7 +85,8 @@ class DiscreteDistributionTest {
Map<String, Long> experimentOutcome = testee.generateRandomStream().limit(1_000_000)
.collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
- assertThat(experimentOutcome.get("a")).isCloseTo(experimentOutcome.get("b"), Offset.offset(5_000L));
+ assertThat(experimentOutcome.get("a"))
+ .isCloseTo(experimentOutcome.get("b"), OFFSET);
}
@Test
@@ -96,7 +98,8 @@ class DiscreteDistributionTest {
Map<String, Long> experimentOutcome = testee.generateRandomStream().limit(1_000_000)
.collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
- assertThat(experimentOutcome.get("a")).isCloseTo(experimentOutcome.get("b") * 2, Offset.offset(5_000L));
+ assertThat(experimentOutcome.get("a"))
+ .isCloseTo(experimentOutcome.get("b") * 2, OFFSET);
}
@Test
@@ -109,7 +112,8 @@ class DiscreteDistributionTest {
Map<String, Long> experimentOutcome = testee.generateRandomStream().limit(1_000_000)
.collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
- assertThat(experimentOutcome.get("a")).isCloseTo(experimentOutcome.get("b") * 2, Offset.offset(5_000L));
+ assertThat(experimentOutcome.get("a"))
+ .isCloseTo(experimentOutcome.get("b") * 2, OFFSET);
}
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 31/31: [ADR] Define quality levels
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4e099d070f9f51c8658578b344770902cce2dc49
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Sun Jun 21 13:27:00 2020 +0700
[ADR] Define quality levels
---
src/adr/0040-quality-levels-definitions.md | 62 ++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/src/adr/0040-quality-levels-definitions.md b/src/adr/0040-quality-levels-definitions.md
new file mode 100644
index 0000000..a4aad52
--- /dev/null
+++ b/src/adr/0040-quality-levels-definitions.md
@@ -0,0 +1,62 @@
+# 40. Quality levels definition
+
+Date: 2020-06-21
+
+## Status
+
+Accepted (lazy consensus)
+
+## Context
+
+We hereby define as an artifact compiled artifact that external people consumes. This includes:
+
+ - libraries
+ - Mail servers
+ - Extensions for James Mail Servers
+ - Command line tools
+
+We designate as a feature an optional, opt-in behaviour of a James server that can be configured by
+user willing to rely on it.
+
+James as a project delivers several artifacts, and features. In order for project users to better
+understand the underlying quality of the artifact they use, as well as the level of risk associated,
+we need to better define some quality levels.
+
+## Decision
+
+For a given artifact or feature, by **mature** we mean that:
+
+ - *interfaces* in components need a contract test suite
+ - *interfaces* have several implementations
+ - *implementation* of these interfaces need to pass this contract test suite which provides unit tests
+ - Decent integration tests coverage is needed
+ - Performance tests need to be conducted out
+ - Quality Assurance with external clients needs to be conducted out
+ - known existing production deployments/usages
+ - usable documentation
+
+This is the maximum quality level delivered by the James project. Users should feel confident using these
+artifacts or features.
+
+By **experimental** we designate an artifact or feature not matching yet the above requirements. However some
+active contributors are willing to raise the quality level of this component, and eventually make it
+mature. Or at least are willing to support users.
+
+Users should have low expectations regarding experimental artifacts or features. They are encouraged to contribute to them
+in order to raise its quality.
+
+By **unsupported** we mean that an artifact or feature do not match most of the *mature* quality conditions. Active
+contributors do not feel confident delivering support for it. This artifact or feature might be deprecated and
+removed from future James releases. Users are strongly encouraged to contribute to the artifact development.
+
+## Consequences
+
+Quality levels need to be mentioned explicitly in the documentation, per artifact, and per feature.
+
+We need to audit existing artifacts and features to make a list of experimental and unsupported artifacts. We will maintain
+JIRA tickets opened from them, detailing for each one of them expected actions to raise the quality level. Maintaining such
+tickets will encourage contributions on experimental and unsupported components.
+
+## References
+
+ - Mailing list discussion: https://www.mail-archive.com/server-dev@james.apache.org/msg66909.html
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 05/31: JAMES-3305 Task manager deserialization
error handling
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit aebfaa90b906e656641f9348844187aa79e44d00
Author: LanKhuat <dl...@linagora.com>
AuthorDate: Tue Jul 14 14:23:43 2020 +0700
JAMES-3305 Task manager deserialization error handling
---
.../distributed/RabbitMQWorkQueue.java | 19 ++--
.../distributed/DistributedTaskManagerTest.java | 122 ++++++++++++++++++++-
.../distributed/RabbitMQWorkQueueTest.java | 5 +-
3 files changed, 133 insertions(+), 13 deletions(-)
diff --git a/server/task/task-distributed/src/main/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueue.java b/server/task/task-distributed/src/main/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueue.java
index f1daf1f..6f5e22e 100644
--- a/server/task/task-distributed/src/main/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueue.java
+++ b/server/task/task-distributed/src/main/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueue.java
@@ -21,6 +21,7 @@
package org.apache.james.task.eventsourcing.distributed;
import static com.rabbitmq.client.MessageProperties.PERSISTENT_TEXT_PLAIN;
+import static org.apache.james.backends.rabbitmq.Constants.REQUEUE;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
@@ -129,13 +130,15 @@ public class RabbitMQWorkQueue implements WorkQueue {
}
private Mono<Task.Result> executeTask(AcknowledgableDelivery delivery) {
- TaskId taskId = TaskId.fromString(delivery.getProperties().getHeaders().get(TASK_ID).toString());
- return Mono.fromCallable(() -> {
- delivery.ack();
- return new String(delivery.getBody(), StandardCharsets.UTF_8);
- }).flatMap(json ->
- deserialize(json, taskId)
- .flatMap(task -> executeOnWorker(taskId, task)));
+ return Mono.fromCallable(() -> TaskId.fromString(delivery.getProperties().getHeaders().get(TASK_ID).toString()))
+ .flatMap(taskId -> deserialize(new String(delivery.getBody(), StandardCharsets.UTF_8), taskId)
+ .doOnNext(task -> delivery.ack())
+ .flatMap(task -> executeOnWorker(taskId, task)))
+ .onErrorResume(error -> {
+ LOGGER.error("Unable to process {} {}", TASK_ID, delivery.getProperties().getHeaders().get(TASK_ID), error);
+ delivery.nack(!REQUEUE);
+ return Mono.empty();
+ });
}
private Mono<Task> deserialize(String json, TaskId taskId) {
@@ -225,4 +228,4 @@ public class RabbitMQWorkQueue implements WorkQueue {
Optional.ofNullable(cancelRequestListenerHandle).ifPresent(Disposable::dispose);
Optional.ofNullable(cancelRequestListener).ifPresent(Receiver::close);
}
-}
+}
\ No newline at end of file
diff --git a/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/DistributedTaskManagerTest.java b/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/DistributedTaskManagerTest.java
index db84283..37e4b7d 100644
--- a/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/DistributedTaskManagerTest.java
+++ b/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/DistributedTaskManagerTest.java
@@ -19,17 +19,27 @@
package org.apache.james.task.eventsourcing.distributed;
+import static com.rabbitmq.client.MessageProperties.PERSISTENT_TEXT_PLAIN;
+import static org.apache.james.backends.cassandra.Scenario.Builder.executeNormally;
+import static org.apache.james.backends.cassandra.Scenario.Builder.fail;
+import static org.apache.james.task.eventsourcing.distributed.RabbitMQWorkQueue.EXCHANGE_NAME;
+import static org.apache.james.task.eventsourcing.distributed.RabbitMQWorkQueue.ROUTING_KEY;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+import static org.awaitility.Duration.FIVE_SECONDS;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
+import java.util.stream.IntStream;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.james.backends.cassandra.CassandraCluster;
import org.apache.james.backends.cassandra.CassandraClusterExtension;
+import org.apache.james.backends.cassandra.Scenario;
import org.apache.james.backends.cassandra.components.CassandraModule;
import org.apache.james.backends.cassandra.init.CassandraZonedDateTimeModule;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule;
@@ -56,6 +66,8 @@ import org.apache.james.server.task.json.dto.TaskDTOModule;
import org.apache.james.server.task.json.dto.TestTaskDTOModules;
import org.apache.james.task.CompletedTask;
import org.apache.james.task.CountDownLatchExtension;
+import org.apache.james.task.FailedTask;
+import org.apache.james.task.FailsDeserializationTask;
import org.apache.james.task.Hostname;
import org.apache.james.task.MemoryReferenceTask;
import org.apache.james.task.Task;
@@ -63,6 +75,7 @@ import org.apache.james.task.TaskExecutionDetails;
import org.apache.james.task.TaskId;
import org.apache.james.task.TaskManager;
import org.apache.james.task.TaskManagerContract;
+import org.apache.james.task.TaskWithId;
import org.apache.james.task.WorkQueue;
import org.apache.james.task.eventsourcing.EventSourcingTaskManager;
import org.apache.james.task.eventsourcing.TaskExecutionDetailsProjection;
@@ -78,9 +91,13 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
+import com.rabbitmq.client.AMQP;
+import reactor.core.publisher.Mono;
+import reactor.rabbitmq.OutboundMessage;
import reactor.rabbitmq.Sender;
class DistributedTaskManagerTest implements TaskManagerContract {
@@ -114,6 +131,9 @@ class DistributedTaskManagerTest implements TaskManagerContract {
static final DTOConverter<TaskExecutionDetails.AdditionalInformation, AdditionalInformationDTO> TASK_ADDITIONAL_INFORMATION_DTO_CONVERTER = DTOConverter.of(ADDITIONAL_INFORMATION_MODULE);
static final Hostname HOSTNAME = new Hostname("foo");
static final Hostname HOSTNAME_2 = new Hostname("bar");
+ static final TaskId TASK_ID = TaskId.fromString("2c7f4081-aa30-11e9-bf6c-2d3b9e84aafd");
+ static final Task TASK = new CompletedTask();
+ static final TaskWithId TASK_WITH_ID = new TaskWithId(TASK_ID, TASK);
@RegisterExtension
static final RabbitMQExtension rabbitMQExtension = RabbitMQExtension.singletonRabbitMQ();
@@ -132,6 +152,7 @@ class DistributedTaskManagerTest implements TaskManagerContract {
ImmutableSet<TaskDTOModule<?, ?>> taskDTOModules =
ImmutableSet.of(
+ TestTaskDTOModules.FAILS_DESERIALIZATION_TASK_MODULE,
TestTaskDTOModules.COMPLETED_TASK_MODULE,
TestTaskDTOModules.FAILED_TASK_MODULE,
TestTaskDTOModules.THROWING_TASK_MODULE,
@@ -337,7 +358,7 @@ class DistributedTaskManagerTest implements TaskManagerContract {
}
@Test
- void givenTwoTaskManagerIfTheFirstOneIsDownTheSecondOneShouldBeAbleToRunTheRemainingTasks(CountDownLatch countDownLatch) throws Exception {
+ void givenTwoTaskManagerIfTheFirstOneIsDownTheSecondOneShouldBeAbleToRunTheRemainingTasks(CountDownLatch countDownLatch) {
try (EventSourcingTaskManager taskManager1 = taskManager();
EventSourcingTaskManager taskManager2 = taskManager(HOSTNAME_2)) {
ImmutableBiMap<EventSourcingTaskManager, Hostname> hostNameByTaskManager = ImmutableBiMap.of(taskManager1, HOSTNAME, taskManager2, HOSTNAME_2);
@@ -364,6 +385,105 @@ class DistributedTaskManagerTest implements TaskManagerContract {
}
}
+ @Test
+ void shouldNotCrashWhenBadMessage() {
+ TaskManager taskManager = taskManager(HOSTNAME);
+
+ taskManager.submit(new FailsDeserializationTask());
+
+ TaskId id = taskManager.submit(TASK);
+
+ awaitUntilTaskHasStatus(id, TaskManager.Status.COMPLETED, taskManager);
+ }
+
+ @Test
+ void shouldNotCrashWhenBadMessages() {
+ TaskManager taskManager = taskManager(HOSTNAME);
+
+ IntStream.range(0, 100).forEach(i -> taskManager.submit(new FailsDeserializationTask()));
+
+ TaskId id = taskManager.submit(TASK);
+
+ awaitUntilTaskHasStatus(id, TaskManager.Status.COMPLETED, taskManager);
+ }
+
+ @Test
+ void shouldNotCrashWhenInvalidHeader() throws Exception {
+ TaskManager taskManager = taskManager(HOSTNAME);
+
+ AMQP.BasicProperties badProperties = new AMQP.BasicProperties.Builder()
+ .deliveryMode(PERSISTENT_TEXT_PLAIN.getDeliveryMode())
+ .priority(PERSISTENT_TEXT_PLAIN.getPriority())
+ .contentType(PERSISTENT_TEXT_PLAIN.getContentType())
+ .headers(ImmutableMap.of("abc", TASK_WITH_ID.getId().asString()))
+ .build();
+
+ rabbitMQExtension.getSender()
+ .send(Mono.just(new OutboundMessage(EXCHANGE_NAME,
+ ROUTING_KEY, badProperties, taskSerializer.serialize(TASK_WITH_ID.getTask()).getBytes(StandardCharsets.UTF_8))))
+ .block();
+
+ TaskId taskId = taskManager.submit(TASK);
+
+ await().atMost(FIVE_SECONDS).until(() -> taskManager.list(TaskManager.Status.COMPLETED).size() == 1);
+
+ assertThat(taskManager.getExecutionDetails(taskId).getStatus())
+ .isEqualTo(TaskManager.Status.COMPLETED);
+ }
+
+ @Test
+ void shouldNotCrashWhenInvalidTaskId() throws Exception {
+ TaskManager taskManager = taskManager(HOSTNAME);
+
+ AMQP.BasicProperties badProperties = new AMQP.BasicProperties.Builder()
+ .deliveryMode(PERSISTENT_TEXT_PLAIN.getDeliveryMode())
+ .priority(PERSISTENT_TEXT_PLAIN.getPriority())
+ .contentType(PERSISTENT_TEXT_PLAIN.getContentType())
+ .headers(ImmutableMap.of("taskId", "BAD_ID"))
+ .build();
+
+ rabbitMQExtension.getSender()
+ .send(Mono.just(new OutboundMessage(EXCHANGE_NAME,
+ ROUTING_KEY, badProperties, taskSerializer.serialize(TASK_WITH_ID.getTask()).getBytes(StandardCharsets.UTF_8))))
+ .block();
+
+ TaskId taskId = taskManager.submit(TASK);
+
+ await().atMost(FIVE_SECONDS).until(() -> taskManager.list(TaskManager.Status.COMPLETED).size() == 1);
+
+ assertThat(taskManager.getExecutionDetails(taskId).getStatus())
+ .isEqualTo(TaskManager.Status.COMPLETED);
+ }
+
+ @Test
+ void shouldNotCrashWhenErrorHandlingFails(CassandraCluster cassandra) throws Exception {
+ TaskManager taskManager = taskManager(HOSTNAME);
+
+ cassandra.getConf().printStatements();
+ cassandra.getConf().registerScenario(Scenario.combine(
+ executeNormally()
+ .times(2) // submit + inProgress
+ .whenQueryStartsWith("INSERT INTO eventStore"),
+ executeNormally()
+ .times(2) // submit + inProgress
+ .whenQueryStartsWith("INSERT INTO taskExecutionDetailsProjection"),
+ fail()
+ .forever()
+ .whenQueryStartsWith("INSERT INTO eventStore"),
+ fail()
+ .forever()
+ .whenQueryStartsWith("INSERT INTO taskExecutionDetailsProjection")));
+ taskManager.submit(new FailedTask());
+
+ Thread.sleep(1000);
+
+ cassandra.getConf().registerScenario(Scenario.NOTHING);
+
+ TaskId id2 = taskManager.submit(new CompletedTask());
+
+ awaitUntilTaskHasStatus(id2, TaskManager.Status.COMPLETED, taskManager);
+ }
+
private Hostname getOtherNode(ImmutableBiMap<EventSourcingTaskManager, Hostname> hostNameByTaskManager, Hostname node) {
return hostNameByTaskManager
.values()
diff --git a/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueueTest.java b/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueueTest.java
index 4eb0323..5832863 100644
--- a/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueueTest.java
+++ b/server/task/task-distributed/src/test/java/org/apache/james/task/eventsourcing/distributed/RabbitMQWorkQueueTest.java
@@ -25,7 +25,6 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.awaitility.Awaitility.await;
import static org.awaitility.Duration.FIVE_HUNDRED_MILLISECONDS;
import static org.awaitility.Duration.TWO_SECONDS;
-import static org.mockito.Mockito.spy;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.IntStream;
@@ -67,7 +66,7 @@ class RabbitMQWorkQueueTest {
@BeforeEach
void setUp() {
- worker = spy(new ImmediateWorker());
+ worker = new ImmediateWorker();
serializer = JsonTaskSerializer.of(TestTaskDTOModules.COMPLETED_TASK_MODULE, TestTaskDTOModules.MEMORY_REFERENCE_TASK_MODULE.apply(new MemoryReferenceTaskStore()));
testee = new RabbitMQWorkQueue(worker, rabbitMQExtension.getSender(), rabbitMQExtension.getReceiverProvider(), serializer);
testee.start();
@@ -160,7 +159,5 @@ class RabbitMQWorkQueueTest {
assertThatThrownBy(() -> await().atMost(FIVE_HUNDRED_MILLISECONDS).untilAtomic(counter, CoreMatchers.equalTo(3L))).isInstanceOf(ConditionTimeoutException.class);
assertThatCode(() -> await().atMost(TWO_SECONDS).untilAtomic(counter, CoreMatchers.equalTo(3L))).doesNotThrowAnyException();
-
}
-
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 27/31: JAMES-3302 Migrate operator guide
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit cd105c8b4e67d3bf543c01e6603ba3051706bec5
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 10 14:25:09 2020 +0700
JAMES-3302 Migrate operator guide
---
.../servers/pages/distributed/operate/guide.adoc | 606 ++++++++++++++++++++-
.../servers/pages/distributed/operate/metrics.adoc | 9 +-
2 files changed, 612 insertions(+), 3 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/operate/guide.adoc b/docs/modules/servers/pages/distributed/operate/guide.adoc
index 87adad0..6f6f921 100644
--- a/docs/modules/servers/pages/distributed/operate/guide.adoc
+++ b/docs/modules/servers/pages/distributed/operate/guide.adoc
@@ -1,4 +1,606 @@
= Operator guide
-(TODO migrate and adapt content from
-https://github.com/linagora/james-project/blob/master/src/site/markdown/server/manage-guice-distributed-james.md)
\ No newline at end of file
+This guide aims to be an entry-point to the James documentation for user
+managing a distributed Guice James server.
+
+It includes:
+
+* Simple architecture explanations
+* Propose some diagnostics for some common issues
+* Present procedures that can be set up to address these issues
+
+In order to not duplicate information, existing documentation will be
+linked.
+
+Please note that this product is under active development, should be
+considered experimental and thus targets advanced users.
+
+== Basic Monitoring
+
+A toolbox is available to help an administrator diagnose issues:
+
+* xref:#logging.adoc[Structured logging into Kibana]
+* link:#metrics.adoc[Metrics graphs into Grafana]
+* xref:manage-webadmin.adoc#_healthcheck[WebAdmin HealthChecks]
+
+== Mail processing
+
+Currently, an administrator can monitor mail processing failure through `ERROR` log
+review. We also recommend watching in Kibana INFO logs using the
+`org.apache.james.transport.mailets.ToProcessor` value as their `logger`. Metrics about
+mail repository size, and the corresponding Grafana boards are yet to be contributed.
+
+Furthermore, given the default mailet container configuration, we recommend monitoring
+`cassandra://var/mail/error/` to be empty.
+
+WebAdmin exposes all utilities for
+xref:manage-webadmin.html#_reprocessing_mails_from_a_mail_repository[reprocessing
+all mails in a mail repository] or
+xref:manage-webadmin.html#_reprocessing_a_specific_mail_from_a_mail_repository[reprocessing
+a single mail in a mail repository].
+
+Also, one can decide to
+xref:manage-webadmin.html#_removing_all_mails_from_a_mail_repository[delete
+all the mails of a mail repository] or
+xref:manage-webadmin.html#_removing_a_mail_from_a_mail_repository[delete
+a single mail of a mail repository].
+
+Performance of mail processing can be monitored via the
+https://github.com/apache/james-project/blob/master/grafana-reporting/MAILET-1490071694187-dashboard.json[mailet
+grafana board] and
+https://github.com/apache/james-project/blob/master/grafana-reporting/MATCHER-1490071813409-dashboard.json[matcher
+grafana board].
+
+=== Recipient rewriting
+
+Given the default configuration, errors (like loops) uopn recipient rewritting will lead
+to emails being stored in `cassandra://var/mail/rrt-error/`.
+
+We recommend monitoring the content of this mail repository to be empty.
+
+If it is not empty, we recommend
+verifying user mappings via xref:webadmin.adoc#_user_mappings[User
+Mappings webadmin API] then once identified break the loop by removing
+some Recipient Rewrite Table entry via the
+xref:webadmin.adoc#_removing_an_alias_of_an_user[Delete Alias],
+xref:webadmin.adoc#_removing_a_group_member[Delete Group member],
+xref:webadmin.adoc#_removing_a_destination_of_a_forward[Delete
+forward], xref:webadmin.adoc#_remove_an_address_mapping[Delete
+Address mapping],
+xref:webadmin.adoc#_removing_a_domain_mapping[Delete Domain
+mapping] or xref:webadmin.adoc#_removing_a_regex_mapping[Delete
+Regex mapping] APIs (as needed).
+
+The `Mail.error` field can help diagnose the issue as well. Then once
+the root cause has been addressed, the mail can be reprocessed.
+
+== Mailbox Event Bus
+
+It is possible for the administrator of James to define the mailbox
+listeners he wants to use, by adding them in the
+https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/listeners.xml[listeners.xml]
+configuration file. It’s possible also to add your own custom mailbox
+listeners. This enables to enhance capabilities of James as a Mail
+Delivery Agent. You can get more information about those
+link:config-listeners.html[here].
+
+Currently, an administrator can monitor listeners failures through
+`ERROR` log review. Metrics regarding mailbox listeners can be monitored
+via
+https://github.com/apache/james-project/blob/master/grafana-reporting/MailboxListeners-1528958667486-dashboard.json[mailbox_listeners
+grafana board] and
+https://github.com/apache/james-project/blob/master/grafana-reporting/MailboxListeners%20rate-1552903378376.json[mailbox_listeners_rate
+grafana board].
+
+Upon exceptions, a bounded number of retries are performed (with
+exponential backoff delays). If after those retries the listener is
+still failing to perform its operation, then the event will be stored in
+the xref:webadmin.adoc#_event_dead_letter[Event Dead Letter]. This
+API allows diagnosing issues, as well as redelivering the events.
+
+To check that you have undelivered events in your system, you can first
+run the associated with
+xref:webadmin.adoc#_healthcheck[event dead letter health check] .
+You can explore Event DeadLetter content through WebAdmin. For
+this, xref:webadmin.adoc#_listing_mailbox_listener_groups[list mailbox listener groups]
+you will get a list of groups back, allowing
+you to check if those contain registered events in each by
+xref:webadmin.adoc#_listing_failed_events[listing their failed events].
+
+If you get failed events IDs back, you can as well
+xref:webadmin.adoc#_getting_event_details[check their details].
+
+An easy way to solve this is just to trigger then the
+xref:webadmin.adoc#_redeliver_all_events[redeliver all events]
+task. It will start reprocessing all the failed events registered in
+event dead letters.
+
+If for some other reason you don’t need to redeliver all events, you
+have more fine-grained operations allowing you to
+xref:webadmin.adoc#_redeliver_group_events[redeliver group events]
+or even just
+xref:webadmin.adoc#_redeliver_a_single_event[redeliver a single event].
+
+== ElasticSearch Indexing
+
+A projection of messages is maintained in ElasticSearch via a listener
+plugged into the mailbox event bus in order to enable search features.
+
+You can find more information about ElasticSearch configuration
+link:config-elasticsearch.html[here].
+
+=== Usual troubleshooting procedures
+
+As explained in the link:#_mailbox_event_bus[Mailbox Event Bus] section,
+processing those events can fail sometimes.
+
+Currently, an administrator can monitor indexation failures through
+`ERROR` log review. You can as well
+xref:manage-webadmin.html#_listing_failed_events[list failed events] by
+looking with the group called
+`org.apache.james.mailbox.elasticsearch.events.ElasticSearchListeningMessageSearchIndex$ElasticSearchListeningMessageSearchIndexGroup`.
+A first on-the-fly solution could be to just
+link:#_mailbox_event_bus[redeliver those group events with event dead letter].
+
+If the event storage in dead-letters fails (for instance in the face of
+Cassandra storage exceptions), then you might need to use our WebAdmin
+reIndexing tasks.
+
+From there, you have multiple choices. You can
+xref:webadmin.adoc#_reIndexing_all_mails[reIndex all mails],
+xref:webadmin.adoc#_reIndexing_a_mailbox_mails[reIndex mails from a mailbox] or even just
+xref:webadmin.adoc#_reIndexing_a_single_mail[reIndex a single mail].
+
+When checking the result of a reIndexing task, you might have failed
+reprocessed mails. You can still use the task ID to
+xref:webadmin.adoc#_fixing_previously_failed_reIndexing[reprocess previously failed reIndexing mails].
+
+=== On the fly ElasticSearch Index setting update
+
+Sometimes you might need to update index settings. Cases when an
+administrator might want to update index settings include:
+
+* Scaling out: increasing the shard count might be needed.
+* Changing string analysers, for instance to target another language
+* etc.
+
+In order to achieve such a procedure, you need to:
+
+* https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-create-index.html[Create
+the new index] with the right settings and mapping
+* James uses two aliases on the mailbox index: one for reading
+(`mailboxReadAlias`) and one for writing (`mailboxWriteAlias`). First
+https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-aliases.html[add
+an alias] `mailboxWriteAlias` to that new index, so that now James
+writes on the old and new indexes, while only keeping reading on the
+first one
+* Now trigger a
+https://www.elastic.co/guide/en/elasticsearch/reference/6.3/docs-reindex.html[reindex]
+from the old index to the new one (this actively relies on `_source`
+field being present)
+* When this is done, add the `mailboxReadAlias` alias to the new index
+* Now that the migration to the new index is done, you can
+https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-delete-index.html[drop
+the old index]
+* You might want as well modify the James configuration file
+https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties[elasticsearch.properties]
+by setting the parameter `elasticsearch.index.mailbox.name` to the name
+of your new index. This is to avoid that James re-creates index upon
+restart
+
+_Note_: keep in mind that reindexing can be a very long operation
+depending on the volume of mails you have stored.
+
+== Solving cassandra inconsistencies
+
+Cassandra backend uses data duplication to workaround Cassandra query
+limitations. However, Cassandra is not doing transaction when writing in
+several tables, this can lead to consistency issues for a given piece of
+data. The consequence could be that the data is in a transient state
+(that should never appear outside of the system).
+
+Because of the lack of transactions, it’s hard to prevent these kind of
+issues. We had developed some features to fix some existing cassandra
+inconsistency issues that had been reported to James.
+
+=== Jmap message fast view projections
+
+When you read a Jmap message, some calculated properties are expected to
+be fast to retrieve, like `preview`, `hasAttachment`. James achieves it
+by pre-calculating and storing them into a caching table
+(`message_fast_view_projection`). Missing caches are populated on
+message reads and will temporary decrease the performance.
+
+==== How to detect the outdated projections
+
+You can watch the `MessageFastViewProjection` health check at
+xref:webadmin.adoc#_check_all_components[webadmin documentation].
+It provides a check based on the ratio of missed projection reads.
+
+==== How to solve
+
+Since the MessageFastViewProjection is self healing, you should be
+concerned only if the health check still returns `degraded` for a while,
+there’s a possible thing you can do is looking at James logs for more
+clues.
+
+=== Mailboxes
+
+`mailboxPath` and `mailbox` tables share common fields like `mailboxId`
+and mailbox `name`. A successful operation of creating/renaming/delete
+mailboxes has to succeed at updating `mailboxPath` and `mailbox` table.
+Any failure on creating/updating/delete records in `mailboxPath` or
+`mailbox` can produce inconsistencies.
+
+==== How to detect the inconsistencies
+
+If you found the suspicious `MailboxNotFoundException` in your logs.
+Currently, there’s no dedicated tool for that, we recommend scheduling
+the SolveInconsistencies task below for the mailbox object on a regular
+basis, avoiding peak traffic in order to address both inconsistencies
+diagnostic and fixes.
+
+==== How to solve
+
+An admin can run offline webadmin
+xref:webadmin.adoc#_fixing_mailboxes_inconsistencies[solve Cassandra mailbox object inconsistencies task]
+in order to sanitize his
+mailbox denormalization.
+
+In order to ensure being offline, stop the traffic on SMTP, JMAP and
+IMAP ports, for example via re-configuration or firewall rules.
+
+=== Mailboxes Counters
+
+James maintains a per mailbox projection for message count and unseen
+message count. Failures during the denormalization process will lead to
+incorrect results being returned.
+
+==== How to detect the inconsistencies
+
+Incorrect message count/message unseen count could be seen in the
+`Mail User Agent` (IMAP or JMAP). Invalid values are reported in the
+logs as warning with the following class
+`org.apache.james.mailbox.model.MailboxCounters` and the following
+message prefix: `Invalid mailbox counters`.
+
+==== How to solve
+
+Execute the
+xref:webadmin.adoc#_recomputing_mailbox_counters[recompute Mailbox counters task].
+This task is not concurrent-safe. Concurrent
+increments & decrements will be ignored during a single mailbox
+processing. Re-running this task may eventually return the correct
+result.
+
+=== Messages
+
+Messages are denormalized and stored in both `imapUidTable` (source of
+truth) and `messageIdTable`. Failure in the denormalization process will
+cause inconsistencies between the two tables.
+
+==== How to detect the inconsistencies
+
+User can see a message in JMAP but not in IMAP, or mark a message as
+`SEEN' in JMAP but the message flag is still unchanged in IMAP.
+
+==== How to solve
+
+Execute the
+xref:webadmin.adoc#_fixing_messages_inconsistencies[solve Cassandra message inconsistencies task]. This task is not
+concurrent-safe. User actions concurrent to the inconsistency fixing
+task could result in new inconsistencies being created. However the
+source of truth `imapUidTable` will not be affected and thus re-running
+this task may eventually fix all issues.
+
+=== Quotas
+
+User can monitor the amount of space and message count he is allowed to
+use, and that he is effectively using. James relies on an event bus and
+Cassandra to track the quota of an user. Upon Cassandra failure, this
+value can be incorrect.
+
+==== How to detect the inconsistencies
+
+Incorrect quotas could be seen in the `Mail User Agent` (IMAP or JMAP).
+
+==== How to solve
+
+Execute the
+xref:webadmin.adoc#_recomputing_current_quotas_for_users[recompute Quotas counters task]. This task is not concurrent-safe. Concurrent
+operations will result in an invalid quota to be persisted. Re-running
+this task may eventually return the correct result.
+
+=== RRT (RecipientRewriteTable) mapping sources
+
+`rrt` and `mappings_sources` tables store information about address
+mappings. The source of truth is `rrt` and `mappings_sources` is the
+projection table containing all mapping sources.
+
+==== How to detect the inconsistencies
+
+Right now there’s no tool for detecting that, we’re proposing a
+https://issues.apache.org/jira/browse/JAMES-3069[development plan]. By
+the mean time, the recommendation is to execute the
+`SolveInconsistencies` task below in a regular basis.
+
+==== How to solve
+
+Execute the Cassandra mapping `SolveInconsistencies` task described in
+xref:webadmin.adoc#_operations_on_mappings_sources[webadmin documentation]
+
+== Setting Cassandra user permissions
+
+When a Cassandra cluster is serving more than a James cluster, the
+keyspaces need isolation. It can be achieved by configuring James server
+with credentials preventing access or modification of other keyspaces.
+
+We recommend you to not use the initial admin user of Cassandra and
+provide a different one with a subset of permissions for each
+application.
+
+=== Prerequisites
+
+We’re gonna use the Cassandra super users to create roles and grant
+permissions for them. To do that, Cassandra requires you to login via
+username/password authentication and enable granting in cassandra
+configuration file.
+
+For example:
+
+....
+echo -e "\nauthenticator: PasswordAuthenticator" >> /etc/cassandra/cassandra.yaml
+echo -e "\nauthorizer: org.apache.cassandra.auth.CassandraAuthorizer" >> /etc/cassandra/cassandra.yaml
+....
+
+=== Prepare Cassandra roles & keyspaces for James
+
+==== Create a role
+
+Have a look at
+http://cassandra.apache.org/doc/3.11.3/cql/security.html[cassandra documentation] section `CREATE ROLE` for more information
+
+E.g.
+
+....
+CREATE ROLE james_one WITH PASSWORD = 'james_one' AND LOGIN = true;
+....
+
+==== Create a keyspace
+
+Have a look at
+http://cassandra.apache.org/doc/3.11.3/cql/ddl.html[cassandra documentation] section `CREATE KEYSPACE` for more information
+
+==== Grant permissions on created keyspace to the role
+
+The role to be used by James needs to have full rights on the keyspace
+that James is using. Assuming the keyspace name is `james_one_keyspace`
+and the role be `james_one`.
+
+....
+GRANT CREATE ON KEYSPACE james_one_keyspace TO james_one; // Permission to create tables on the appointed keyspace
+GRANT SELECT ON KEYSPACE james_one_keyspace TO james_one; // Permission to select from tables on the appointed keyspace
+GRANT MODIFY ON KEYSPACE james_one_keyspace TO james_one; // Permission to update data in tables on the appointed keyspace
+....
+
+*Warning*: The granted role doesn’t have the right to create keyspaces,
+thus, if you haven’t created the keyspace, James server will fail to
+start is expected.
+
+*Tips*
+
+Since all of Cassandra roles used by different James are supposed to
+have a same set of permissions, you can reduce the works by creating a
+base role set like `typical_james_role` with all of necessary
+permissions. After that, with each James, create a new role and grant
+the `typical_james_role` to the newly created one. Note that, once a
+base role set is updated ( granting or revoking rights) all granted
+roles are automatically updated.
+
+E.g.
+
+....
+CREATE ROLE james1 WITH PASSWORD = 'james1' AND LOGIN = true;
+GRANT typical_james_role TO james1;
+
+CREATE ROLE james2 WITH PASSWORD = 'james2' AND LOGIN = true;
+GRANT typical_james_role TO james2;
+....
+
+==== Revoke harmful permissions from the created role
+
+We want a specific role that cannot describe or query the information of
+other keyspaces or tables used by another application. By default,
+Cassandra allows every role created to have the right to describe any
+keyspace and table. There’s no configuration that can make effect on
+that topic. Consequently, you have to accept that your data models are
+still being exposed to anyone having credentials to Cassandra.
+
+For more information, have a look at
+http://cassandra.apache.org/doc/3.11.3/cql/security.html[cassandra documentation] section `REVOKE PERMISSION`.
+
+Except for the case above, the permissions are not auto available for a
+specific role unless they are granted by `GRANT` command. Therefore, if
+you didn’t provide more permissions than
+link:#Grant_permissions_on_created_keyspace_to_the_role[granting
+section], there’s no need to revoke.
+
+== Cassandra table level configuration
+
+While _Distributed James_ is shipped with default table configuration
+options, these settings should be refined depending of your usage.
+
+These options are:
+
+* The https://cassandra.apache.org/doc/latest/operating/compaction.html[compaction algorithms]
+- The https://cassandra.apache.org/doc/latest/operating/bloom_filters.html[bloom filter sizing]
+- The https://cassandra.apache.org/doc/latest/operating/compression.html?highlight=chunk%20size[chunk size]
+- Thehttps://www.datastax.com/blog/2011/04/maximizing-cache-benefit-cassandra[cachingoptions]
+
+The compaction algorithms allow a tradeoff between background IO upon
+writes and reads. We recommend: - Using *Leveled Compaction Strategy* on
+read intensive tables subject to updates. This limits the count of
+SStables being read at the cost of more background IO. High garbage
+collections can be caused by an inappropriate use of Leveled Compaction
+Strategy. - Otherwise use the default *Size Tiered Compaction Strategy*.
+
+Bloom filters help avoiding unnecessary reads on SSTables. This
+probabilistic data structure can tell an entry absence from a SSTable,
+as well as the presence of an entry with an associated probability. If a
+lot of false positives are noticed, the size of the bloom filters can be
+increased.
+
+As explained in
+https://thelastpickle.com/blog/2018/08/08/compression_performance.html[this post],
+chunk size used upon compression allows a tradeoff between reads
+and writes. A smaller size will mean decreasing compression, thus it
+increases data being stored on disk, but allow lower chunks to be read
+to access data, and will favor reads. A bigger size will mean better
+compression, thus writing less, but it might imply reading bigger
+chunks.
+
+Cassandra enables a key cache and a row cache. Key cache enables to skip
+reading the partition index upon reads, thus performing 1 read to the
+disk instead of 2. Enabling this cache is globally advised. Row cache
+stores the entire row in memory. It can be seen as an optimization, but
+it might actually use memory no longer available for instance for file
+system cache. We recommend turning it off on modern SSD hardware.
+
+A review of your usage can be conducted using
+https://cassandra.apache.org/doc/latest/tools/nodetool/nodetool.html[nodetool]
+utility. For example `nodetool tablestats {keyspace}` allows reviewing
+the number of SSTables, the read/write ratios, bloom filter efficiency.
+`nodetool tablehistograms {keyspace}.{table}` might give insight about
+read/write performance.
+
+Table level options can be changed using *ALTER TABLE* for example with
+the https://cassandra.apache.org/doc/latest/tools/cqlsh.html[cqlsh]
+utility. A full compaction might be needed in order for the changes to
+be taken into account.
+
+== Mail Queue
+
+=== Fine tune configuration for RabbitMQ
+
+In order to adapt mail queue settings to the actual traffic load, an
+administrator needs to perform fine configuration tunning as explained
+in
+https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-rabbitmq.xml[rabbitmq.properties].
+
+Be aware that `MailQueue::getSize` is currently performing a browse and
+thus is expensive. Size recurring metric reporting thus introduces
+performance issues. As such, we advise setting
+`mailqueue.size.metricsEnabled=false`.
+
+=== Managing email queues
+
+Managing an email queue is an easy task if you follow this procedure:
+
+* First, xref:webadmin.adoc#_listing_mail_queues[List mail queues]
+and xref:webadmin.adoc#Getting_a_mail_queue_details[get a mail
+queue details].
+* And then
+xref:webadmin.adoc#_listing_the_mails_of_a_mail_queue[List the mails of a mail queue].
+
+In case, you need to clear an email queue because there are only spam or
+trash emails in the email queue you have this procedure to follow:
+
+* All mails from the given mail queue will be deleted with
+xref:webadmin.adoc#_clearing_a_mail_queue[Clearing a mail queue].
+
+== Updating Cassandra schema version
+
+A schema version indicates you which schema your James server is relying
+on. The schema version number tracks if a migration is required. For
+instance, when the latest schema version is 2, and the current schema
+version is 1, you might think that you still have data in the deprecated
+Message table in the database. Hence, you need to migrate these messages
+into the MessageV2 table. Once done, you can safely bump the current
+schema version to 2.
+
+Relying on outdated schema version prevents you to benefit from the
+newest performance and safety improvements. Otherwise, there’s something
+very unexpected in the way we manage cassandra schema: we create new
+tables without asking the admin about it. That means your James version
+is always using the last tables but may also take into account the old
+ones if the migration is not done yet.
+
+=== How to detect when we should update Cassandra schema version
+
+When you see in James logs
+`org.apache.james.modules.mailbox.CassandraSchemaVersionStartUpCheck`
+showing a warning like `Recommended version is versionX`, you should
+perform an update of the Cassandra schema version.
+
+Also, we keep track of changes needed when upgrading to a newer version.
+You can read this
+https://github.com/apache/james-project/blob/master/upgrade-instructions.md[upgrade
+instructions].
+
+=== How to update Cassandra schema version
+
+These schema updates can be triggered by webadmin using the Cassandra
+backend. Following steps are for updating Cassandra schema version:
+
+* At the very first step, you need to
+xref:webadmin.adoc#_retrieving_current_cassandra_schema_version[retrieve
+current Cassandra schema version]
+* And then, you
+xref:webadmin.adoc#_retrieving_latest_available_cassandra_schema_version[retrieve
+latest available Cassandra schema version] to make sure there is a
+latest available version
+* Eventually, you can update the current schema version to the one you
+got with
+xref:webadmin.adoc#_upgrading_to_the_latest_version[upgrading to
+the latest version]
+
+Otherwise, if you need to run the migrations to a specific version, you
+can use
+xref:webadmin.adoc#_upgrading_to_a_specific_version[Upgrading to a
+specific version]
+
+== Deleted Message Vault
+
+We recommend the administrator to
+xref:#_cleaning_expired_deleted_messages[run it] in cron job to save
+storage volume.
+
+=== How to configure deleted messages vault
+
+To setup James with Deleted Messages Vault, you need to follow those
+steps:
+
+* Enable Deleted Messages Vault by configuring Pre Deletion Hooks.
+* Configuring the retention time for the Deleted Messages Vault.
+
+==== Enable Deleted Messages Vault by configuring Pre Deletion Hooks
+
+You need to configure this hook in
+https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/listeners.xml[listeners.xml]
+configuration file. More details about configuration & example can be
+found at http://james.apache.org/server/config-listeners.html[Pre
+Deletion Hook Configuration]
+
+==== Configuring the retention time for the Deleted Messages Vault
+
+In order to configure the retention time for the Deleted Messages Vault,
+an administrator needs to perform fine configuration tunning as
+explained in
+https://github.com/apache/james-project/blob/master/dockerfiles/run/guice/cassandra/destination/conf/deletedMessageVault.properties[deletedMessageVault.properties].
+Mails are not retained forever as you have to configure a retention
+period (by `retentionPeriod`) before using it (with one-year retention
+by default if not defined).
+
+=== Restore deleted messages after deletion
+
+After users deleted their mails and emptied the trash, the admin can use
+xref:webadmin.adoc#_deleted-messages-vault[Restore Deleted Messages]
+to restore all the deleted mails.
+
+=== Cleaning expired deleted messages
+
+You can delete all deleted messages older than the configured
+`retentionPeriod` by using
+xref:webadmin.adoc#_deleted-messages-vault[Purge Deleted Messages].
+We recommend calling this API in CRON job on 1st day each
+month.
diff --git a/docs/modules/servers/pages/distributed/operate/metrics.adoc b/docs/modules/servers/pages/distributed/operate/metrics.adoc
index e46ca99..f070b2b 100644
--- a/docs/modules/servers/pages/distributed/operate/metrics.adoc
+++ b/docs/modules/servers/pages/distributed/operate/metrics.adoc
@@ -1,4 +1,11 @@
= Metrics
(TODO migrate and adapt content from
-https://github.com/linagora/james-project/blob/master/src/site/xdoc/server/metrics.xml)
\ No newline at end of file
+https://github.com/linagora/james-project/blob/master/src/site/xdoc/server/metrics.xml)
+
+If some metrics seem abnormally slow despite in depth database
+performance tuning, feedback is appreciated as well on the bug tracker,
+the user mailing list or our Gitter channel (see our
+http://james.apache.org/#second[community page]) . Any additional
+details categorizing the slowness are appreciated as well (details of
+the slow requests for instance).
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 03/31: JAMES-3107 Log slow traces to WARN
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 85ba240856715e7c35bce354e2227c980e2c8d55
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 10:15:02 2020 +0700
JAMES-3107 Log slow traces to WARN
---
.../java/org/apache/james/metrics/dropwizard/DropWizardTimeMetric.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/metrics/metrics-dropwizard/src/main/java/org/apache/james/metrics/dropwizard/DropWizardTimeMetric.java b/metrics/metrics-dropwizard/src/main/java/org/apache/james/metrics/dropwizard/DropWizardTimeMetric.java
index 834da92..ab0fe78 100644
--- a/metrics/metrics-dropwizard/src/main/java/org/apache/james/metrics/dropwizard/DropWizardTimeMetric.java
+++ b/metrics/metrics-dropwizard/src/main/java/org/apache/james/metrics/dropwizard/DropWizardTimeMetric.java
@@ -57,7 +57,7 @@ public class DropWizardTimeMetric implements TimeMetric {
public ExecutionResult logWhenExceedP99(Duration thresholdInNanoSeconds) {
Preconditions.checkNotNull(thresholdInNanoSeconds);
if (elasped.compareTo(p99) > 0 && elasped.compareTo(thresholdInNanoSeconds) > 0) {
- LOGGER.info("{} metrics took {} nano seconds to complete, exceeding its {} nano seconds p99",
+ LOGGER.warn("{} metrics took {} nano seconds to complete, exceeding its {} nano seconds p99",
name, elasped, p99);
}
return this;
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 23/31: JAMES-3302 Migrate WebAdmin documentation to
Antora
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 466973978fca0361fbc899b33a7abe1ed29fb9d4
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 9 13:52:27 2020 +0700
JAMES-3302 Migrate WebAdmin documentation to Antora
Adapted to remove 'only on some products' mentions.
Note that some liks needs to be fixed.
---
.../pages/distributed/operate/webadmin.adoc | 4047 +++++++++++++++++++-
.../servers/pages/distributed/run-docker.adoc | 38 +-
2 files changed, 4079 insertions(+), 6 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/operate/webadmin.adoc b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
index ed1b75a..55c589d 100644
--- a/docs/modules/servers/pages/distributed/operate/webadmin.adoc
+++ b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
@@ -1,4 +1,4047 @@
= WebAdmin REST administration API
-(TODO migrate and adapt content from
-https://github.com/linagora/james-project/blob/master/src/site/markdown/server/manage-webadmin.md)
\ No newline at end of file
+The web administration supports for now the CRUD operations on the domains, the users, their mailboxes and their quotas,
+ managing mail repositories, performing cassandra migrations, and much more, as described in the following sections.
+
+*WARNING*: This API allow authentication only via the use of JWT. If not
+configured with JWT, an administrator should ensure an attacker can not
+use this API.
+
+By the way, some endpoints are not filtered by authentication. Those endpoints are not related to data stored in James,
+for example: Swagger documentation & James health checks.
+
+In case of any error, the system will return an error message which is
+json format like this:
+
+....
+{
+ statusCode: <error_code>,
+ type: <error_type>,
+ message: <the_error_message>
+ cause: <the_detail_message_from_throwable>
+}
+....
+
+Also be aware that, in case things go wrong, all endpoints might return
+a 500 internal error (with a JSON body formatted as exposed above). To
+avoid information duplication, this is omitted on endpoint specific
+documentation.
+
+Finally, please note that in case of a malformed URL the 400 bad request
+response will contain an HTML body.
+
+== HealthCheck
+
+=== Check all components
+
+This endpoint is simple for now and is just returning the http status
+code corresponding to the state of checks (see below). The user has to
+check in the logs in order to have more information about failing
+checks.
+
+....
+curl -XGET http://ip:port/healthcheck
+....
+
+Will return a list of healthChecks execution result, with an aggregated
+result:
+
+....
+{
+ "status": "healthy",
+ "checks": [
+ {
+ "componentName": "Cassandra backend",
+ "escapedComponentName": "Cassandra%20backend",
+ "status": "healthy"
+ "cause": null
+ }
+ ]
+}
+....
+
+*status* field can be:
+
+* *healthy*: Component works normally
+* *degraded*: Component works in degraded mode. Some non-critical
+services may not be working, or latencies are high, for example. Cause
+contains explanations.
+* *unhealthy*: The component is currently not working. Cause contains
+explanations.
+
+Supported health checks include:
+
+* *Cassandra backend*: Cassandra storage.
+* *ElasticSearch Backend*: ElasticSearch storage.
+* *EventDeadLettersHealthCheck*
+* *Guice application lifecycle*
+* *JPA Backend*: JPA storage.
+* *MessageFastViewProjection* Health check of the component storing JMAP properties
+which are fast to retrieve. Those properties are computed in advance
+from messages and persisted in order to archive a better performance.
+There are some latencies between a source update and its projections
+updates. Incoherency problems arise when reads are performed in this
+time-window. We piggyback the projection update on missed JMAP read in
+order to decrease the outdated time window for a given entry. The health
+is determined by the ratio of missed projection reads. (lower than 10%
+causes `degraded`)
+* *RabbitMQ backend*: RabbitMQ messaging.
+
+Response codes:
+
+* 200: All checks have answered with a Healthy or Degraded status. James
+services can still be used.
+* 503: At least one check have answered with a Unhealthy status
+
+=== Check single component
+
+Performs a health check for the given component. The component is
+referenced by its URL encoded name.
+
+....
+curl -XGET http://ip:port/healthcheck/checks/Cassandra%20backend
+....
+
+Will return the component’s name, the component’s escaped name, the
+health status and a cause.
+
+....
+{
+ "componentName": "Cassandra backend",
+ "escapedComponentName": "Cassandra%20backend",
+ "status": "healthy"
+ "cause": null
+}
+....
+
+Response codes:
+
+* 200: The check has answered with a Healthy or Degraded status.
+* 404: A component with the given name was not found.
+* 503: The check has anwered with a Unhealthy status.
+
+=== List all health checks
+
+This endpoint lists all the available health checks.
+
+....
+curl -XGET http://ip:port/healthcheck/checks
+....
+
+Will return the list of all available health checks.
+
+....
+[
+ {
+ "componentName": "Cassandra backend",
+ "escapedComponentName": "Cassandra%20backend"
+ }
+]
+....
+
+Response codes:
+
+* 200: List of available health checks
+
+== Administrating domains
+
+=== Create a domain
+
+....
+curl -XPUT http://ip:port/domains/domainToBeCreated
+....
+
+Resource name domainToBeCreated:
+
+* can not be null or empty
+* can not contain `@'
+* can not be more than 255 characters
+* can not contain `/'
+
+Response codes:
+
+* 204: The domain was successfully added
+* 400: The domain name is invalid
+
+=== Delete a domain
+
+....
+curl -XDELETE http://ip:port/domains/{domainToBeDeleted}
+....
+
+Note: Deletion of an auto-detected domain, default domain or of an
+auto-detected ip is not supported. We encourage you instead to review
+your https://james.apache.org/server/config-domainlist.html[domain list
+configuration].
+
+Response codes:
+
+* 204: The domain was successfully removed
+
+=== Test if a domain exists
+
+....
+curl -XGET http://ip:port/domains/{domainName}
+....
+
+Response codes:
+
+* 204: The domain exists
+* 404: The domain does not exist
+
+=== Get the list of domains
+
+....
+curl -XGET http://ip:port/domains
+....
+
+Possible response:
+
+....
+["domain1", "domain2"]
+....
+
+Response codes:
+
+* 200: The domain list was successfully retrieved
+
+=== Get the list of aliases for a domain
+
+....
+curl -XGET http://ip:port/domains/destination.domain.tld/aliases
+....
+
+Possible response:
+
+....
+[
+ {"source": "source1.domain.tld"},
+ {"source": "source2.domain.tld"}
+]
+....
+
+When sending an email to an email address having `source1.domain.tld` or
+`source2.domain.tld` as a domain part (example:
+`user@source1.domain.tld`), then the domain part will be rewritten into
+destination.domain.tld (so into `user@destination.domain.tld`).
+
+Response codes:
+
+* 200: The domain aliases was successfully retrieved
+* 400: destination.domain.tld has an invalid syntax
+* 404: destination.domain.tld is not part of handled domains and does
+not have local domains as aliases.
+
+=== Create an alias for a domain
+
+To create a domain alias execute the following query:
+
+....
+curl -XPUT http://ip:port/domains/destination.domain.tld/aliases/source.domain.tld
+....
+
+When sending an email to an email address having `source.domain.tld` as
+a domain part (example: `user@source.domain.tld`), then the domain part
+will be rewritten into `destination.domain.tld` (so into
+`user@destination.domain.tld`).
+
+Response codes:
+
+* 204: The redirection now exists
+* 400: `source.domain.tld` or `destination.domain.tld` have an invalid
+syntax
+* 400: `source, domain` and `destination domain` are the same
+* 404: `source.domain.tld` are not part of handled domains.
+
+=== Delete an alias for a domain
+
+To delete a domain alias execute the following query:
+
+....
+curl -XDELETE http://ip:port/domains/destination.domain.tld/aliases/source.domain.tld
+....
+
+When sending an email to an email address having `source.domain.tld` as
+a domain part (example: `user@source.domain.tld`), then the domain part
+will be rewritten into `destination.domain.tld` (so into
+`user@destination.domain.tld`).
+
+Response codes:
+
+* 204: The redirection now no longer exists
+* 400: `source.domain.tld` or destination.domain.tld have an invalid
+syntax
+* 400: source, domain and destination domain are the same
+* 404: `source.domain.tld` are not part of handled domains.
+
+== Administrating users
+
+=== Create a user
+
+....
+curl -XPUT http://ip:port/users/usernameToBeUsed \
+ -d '{"password":"passwordToBeUsed"}' \
+ -H "Content-Type: application/json"
+....
+
+Resource name usernameToBeUsed representing valid users, hence it should
+match the criteria at (TODO) link:/server/config-users.html[User Repositories
+documentation]
+
+Response codes:
+
+* 204: The user was successfully created
+* 400: The user name or the payload is invalid
+
+Note: if the user exists already, its password will be updated.
+
+=== Testing a user existence
+
+....
+curl -XHEAD http://ip:port/users/usernameToBeUsed
+....
+
+Resource name ``usernameToBeUsed'' represents a valid user, hence it
+should match the criteria at (TODO) link:/server/config-users.html[User
+Repositories documentation]
+
+Response codes:
+
+* 200: The user exists
+* 400: The user name is invalid
+* 404: The user does not exist
+
+=== Updating a user password
+
+Same than Create, but a user need to exist.
+
+If the user do not exist, then it will be created.
+
+=== Deleting a user
+
+....
+curl -XDELETE http://ip:port/users/{userToBeDeleted}
+....
+
+Response codes:
+
+* 204: The user was successfully deleted
+
+=== Retrieving the user list
+
+....
+curl -XGET http://ip:port/users
+....
+
+The answer looks like:
+
+....
+[{"username":"username@domain-jmapauthentication.tld"},{"username":"username@domain.tld"}]
+....
+
+Response codes:
+
+* 200: The user name list was successfully retrieved
+
+=== Retrieving the list of allowed `From` headers for a given user
+
+This endpoint allows to know which From headers a given user is allowed to use when sending mails.
+
+....
+curl -XGET http://ip:port/users/givenUser/allowedFromHeaders
+....
+
+The answer looks like:
+
+....
+["user@domain.tld","alias@domain.tld"]
+....
+
+Response codes:
+
+* 200: The list was successfully retrieved
+* 400: The user is invalid
+* 404: The user is unknown
+
+== Administrating mailboxes
+
+=== All mailboxes
+
+Several actions can be performed on the server mailboxes.
+
+Request pattern is:
+
+....
+curl -XPOST /mailboxes?action={action1},...
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+The kind of task scheduled depends on the action parameter. See below
+for details.
+
+==== Fixing mailboxes inconsistencies
+
+....
+curl -XPOST /mailboxes?task=SolveInconsistencies
+....
+
+Will schedule a task for fixing inconsistencies for the mailbox
+deduplicated object stored in Cassandra.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+The `I-KNOW-WHAT-I-M-DOING` header is mandatory (you can read more
+information about it in the warning section below).
+
+The scheduled task will have the following type
+`solve-mailbox-inconsistencies` and the following
+`additionalInformation`:
+
+....
+{
+ "type":"solve-mailbox-inconsistencies",
+ "processedMailboxEntries": 3,
+ "processedMailboxPathEntries": 3,
+ "fixedInconsistencies": 2,
+ "errors": 1,
+ "conflictingEntries":[{
+ "mailboxDaoEntry":{
+ "mailboxPath":"#private:user:mailboxName",
+ "mailboxId":"464765a0-e4e7-11e4-aba4-710c1de3782b"
+ }," +
+ "mailboxPathDaoEntry":{
+ "mailboxPath":"#private:user:mailboxName2",
+ "mailboxId":"464765a0-e4e7-11e4-aba4-710c1de3782b"
+ }
+ }]
+}
+....
+
+Note that conflicting entry inconsistencies will not be fixed and will
+require to explicitly use link:#correcting-ghost-mailbox[ghost mailbox]
+endpoint in order to merge the conflicting mailboxes and prevent any
+message loss.
+
+*WARNING*: this task can cancel concurrently running legitimate user
+operations upon dirty read. As such this task should be run offline.
+
+A dirty read is when data is read between the two writes of the
+denormalization operations (no isolation).
+
+In order to ensure being offline, stop the traffic on SMTP, JMAP and
+IMAP ports, for example via re-configuration or firewall rules.
+
+Due to all of those risks, a `I-KNOW-WHAT-I-M-DOING` header should be
+positioned to `ALL-SERVICES-ARE-OFFLINE` in order to prevent accidental
+calls.
+
+==== Recomputing mailbox counters
+
+....
+curl -XPOST /mailboxes?task=RecomputeMailboxCounters
+....
+
+Will recompute counters (unseen & total count) for the mailbox object
+stored in Cassandra.
+
+Cassandra maintains a per mailbox projection for message count and
+unseen message count. As with any projection, it can go out of sync,
+leading to inconsistent results being returned to the client.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+The scheduled task will have the following type
+`recompute-mailbox-counters` and the following `additionalInformation`:
+
+....
+{
+ "type":"recompute-mailbox-counters",
+ "processedMailboxes": 3,
+ "failedMailboxes": ["464765a0-e4e7-11e4-aba4-710c1de3782b"]
+}
+....
+
+Note that conflicting inconsistencies entries will not be fixed and will
+require to explicitly use link:#correcting-ghost-mailbox[ghost mailbox]
+endpoint in order to merge the conflicting mailboxes and prevent any
+message loss.
+
+*WARNING*: this task do not take into account concurrent modifications
+upon a single mailbox counter recomputation. Rerunning the task will
+_eventually_ provide the consistent result. As such we advise to run
+this task offline.
+
+In order to ensure being offline, stop the traffic on SMTP, JMAP and
+IMAP ports, for example via re-configuration or firewall rules.
+
+`trustMessageProjection` query parameter can be set to `true`. Content
+of `messageIdTable` (listing messages by their mailbox context) table
+will be trusted and not compared against content of `imapUidTable` table
+(listing messages by their messageId mailbox independent identifier).
+This will result in a better performance running the task at the cost of
+safety in the face of message denormalization inconsistencies.
+
+Defaults to false, which generates additional checks. You can read
+https://github.com/apache/james-project/blob/master/src/adr/0022-cassandra-message-inconsistency.md[this
+ADR] to better understand the message projection and how it can become
+inconsistent.
+
+==== Recomputing Global JMAP fast message view projection
+
+Message fast view projection stores message properties expected to be
+fast to fetch but are actually expensive to compute, in order for
+GetMessages operation to be fast to execute for these properties.
+
+These projection items are asynchronously computed on mailbox events.
+
+You can force the full projection recomputation by calling the following
+endpoint:
+
+....
+curl -XPOST /mailboxes?task=recomputeFastViewProjectionItems
+....
+
+Will schedule a task for recomputing the fast message view projection
+for all mailboxes.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed, per
+second. Defaults to 10.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameters.
+
+Example:
+
+....
+curl -XPOST /mailboxes?task=recomputeFastViewProjectionItems&messagesPerSecond=20
+....
+
+The scheduled task will have the following type
+`RecomputeAllFastViewProjectionItemsTask` and the following
+`additionalInformation`:
+
+....
+{
+ "type":"RecomputeAllPreviewsTask",
+ "processedUserCount": 3,
+ "processedMessageCount": 3,
+ "failedUserCount": 2,
+ "failedMessageCount": 1,
+ "runningOptions": {
+ "messagesPerSecond":20
+ }
+}
+....
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+==== ReIndexing action
+
+Be also aware of the limits of this API:
+
+Warning: During the re-indexing, the result of search operations might
+be altered.
+
+Warning: Canceling this task should be considered unsafe as it will
+leave the currently reIndexed mailbox as partially indexed.
+
+Warning: While we have been trying to reduce the inconsistency window to
+a maximum (by keeping track of ongoing events), concurrent changes done
+during the reIndexing might be ignored.
+
+===== ReIndexing all mails
+
+....
+curl -XPOST http://ip:port/mailboxes?task=reIndex
+....
+
+Will schedule a task for reIndexing all the mails stored on this James
+server.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed per
+second. Default is 50.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameter.
+
+An admin can also specify the reindexing mode it wants to use when
+running the task:
+
+* `mode` the reindexing mode used. There are 2 modes for the moment:
+** `rebuildAll` allows to rebuild all indexes. This is the default mode.
+** `fixOutdated` will check for outdated indexed document and reindex
+only those.
+
+This optional parameter must be passed as query parameter.
+
+It’s good to note as well that there is a limitation with the
+`fixOutdated` mode. As we first collect metadata of stored messages to
+compare them with the ones in the index, a failed `expunged` operation
+might not be well corrected (as the message might not exist anymore but
+still be indexed).
+
+Example:
+
+ curl -XPOST http://ip:port/mailboxes?task=reIndex&messagesPerSecond=200&mode=rebuildAll
+
+The scheduled task will have the following type `full-reindexing` and
+the following `additionalInformation`:
+
+....
+{
+ "type":"full-reindexing",
+ "runningOptions":{
+ "messagesPerSecond":200,
+ "mode":"REBUILD_ALL"
+ },
+ "successfullyReprocessedMailCount":18,
+ "failedReprocessedMailCount": 3,
+ "mailboxFailures": ["12", "23" ],
+ "messageFailures": [
+ {
+ "mailboxId": "1",
+ "uids": [1, 36]
+ }]
+}
+....
+
+===== Fixing previously failed ReIndexing
+
+Will schedule a task for reIndexing all the mails which had failed to be
+indexed from the ReIndexingAllMails task.
+
+Given `bbdb69c9-082a-44b0-a85a-6e33e74287a5` being a `taskId` generated
+for a reIndexing tasks
+
+....
+curl -XPOST 'http://ip:port/mailboxes?task=reIndex&reIndexFailedMessagesOf=bbdb69c9-082a-44b0-a85a-6e33e74287a5'
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed per
+second. Default is 50.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameter.
+
+An admin can also specify the reindexing mode it wants to use when
+running the task:
+
+* `mode` the reindexing mode used. There are 2 modes for the moment:
+** `rebuildAll` allows to rebuild all indexes. This is the default mode.
+** `fixOutdated` will check for outdated indexed document and reindex
+only those.
+
+This optional parameter must be passed as query parameter.
+
+It’s good to note as well that there is a limitation with the
+`fixOutdated` mode. As we first collect metadata of stored messages to
+compare them with the ones in the index, a failed `expunged` operation
+might not be well corrected (as the message might not exist anymore but
+still be indexed).
+
+Example:
+
+....
+curl -XPOST http://ip:port/mailboxes?task=reIndex&reIndexFailedMessagesOf=bbdb69c9-082a-44b0-a85a-6e33e74287a5&messagesPerSecond=200&mode=rebuildAll
+....
+
+The scheduled task will have the following type
+`error-recovery-indexation` and the following `additionalInformation`:
+
+....
+{
+ "type":"error-recovery-indexation"
+ "runningOptions":{
+ "messagesPerSecond":200,
+ "mode":"REBUILD_ALL"
+ },
+ "successfullyReprocessedMailCount":18,
+ "failedReprocessedMailCount": 3,
+ "mailboxFailures": ["12", "23" ],
+ "messageFailures": [{
+ "mailboxId": "1",
+ "uids": [1, 36]
+ }]
+}
+....
+
+== Single mailbox
+
+=== ReIndexing a mailbox mails
+
+....
+curl -XPOST http://ip:port/mailboxes/{mailboxId}?task=reIndex
+....
+
+Will schedule a task for reIndexing all the mails in one mailbox.
+
+Note that `mailboxId' path parameter needs to be a (implementation
+dependent) valid mailboxId.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed per
+second. Default is 50.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameter.
+
+An admin can also specify the reindexing mode it wants to use when
+running the task:
+
+* `mode` the reindexing mode used. There are 2 modes for the moment:
+** `rebuildAll` allows to rebuild all indexes. This is the default mode.
+** `fixOutdated` will check for outdated indexed document and reindex
+only those.
+
+This optional parameter must be passed as query parameter.
+
+It’s good to note as well that there is a limitation with the
+`fixOutdated` mode. As we first collect metadata of stored messages to
+compare them with the ones in the index, a failed `expunged` operation
+might not be well corrected (as the message might not exist anymore but
+still be indexed).
+
+Example:
+
+....
+curl -XPOST http://ip:port/mailboxes/{mailboxId}?task=reIndex&messagesPerSecond=200&mode=fixOutdated
+....
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+The scheduled task will have the following type `mailbox-reindexing` and
+the following `additionalInformation`:
+
+....
+{
+ "type":"mailbox-reindexing",
+ "runningOptions":{
+ "messagesPerSecond":200,
+ "mode":"FIX_OUTDATED"
+ },
+ "mailboxId":"{mailboxId}",
+ "successfullyReprocessedMailCount":18,
+ "failedReprocessedMailCount": 3,
+ "mailboxFailures": ["12"],
+ "messageFailures": [
+ {
+ "mailboxId": "1",
+ "uids": [1, 36]
+ }]
+}
+....
+
+Warning: During the re-indexing, the result of search operations might
+be altered.
+
+Warning: Canceling this task should be considered unsafe as it will
+leave the currently reIndexed mailbox as partially indexed.
+
+Warning: While we have been trying to reduce the inconsistency window to
+a maximum (by keeping track of ongoing events), concurrent changes done
+during the reIndexing might be ignored.
+
+== Administrating Messages
+
+=== ReIndexing a single mail by messageId
+
+....
+curl -XPOST http://ip:port/messages/{messageId}?task=reIndex
+....
+
+Will schedule a task for reIndexing a single email in all the mailboxes
+containing it.
+
+Note that `messageId' path parameter needs to be a (implementation
+dependent) valid messageId.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+The scheduled task will have the following type `messageId-reindexing`
+and the following `additionalInformation`:
+
+....
+{
+ "messageId":"18"
+}
+....
+
+Warning: During the re-indexing, the result of search operations might
+be altered.
+
+=== Fixing message inconsistencies
+
+This task is only available on top of Guice Cassandra products.
+
+....
+curl -XPOST /messages?task=SolveInconsistencies
+....
+
+Will schedule a task for fixing message inconsistencies created by the
+message denormalization process.
+
+Messages are denormalized and stored in separated data tables in
+Cassandra, so they can be accessed by their unique identifier or mailbox
+identifier & local mailbox identifier through different protocols.
+
+Failure in the denormalization process will lead to inconsistencies, for
+example:
+
+....
+BOB receives a message
+The denormalization process fails
+BOB can read the message via JMAP
+BOB cannot read the message via IMAP
+
+BOB marks a message as SEEN
+The denormalization process fails
+The message is SEEN via JMAP
+The message is UNSEEN via IMAP
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate of messages to be processed per second.
+Default is 100.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameter.
+
+An admin can also specify the reindexing mode it wants to use when
+running the task:
+
+* `mode` the reindexing mode used. There are 2 modes for the moment:
+** `rebuildAll` allows to rebuild all indexes. This is the default mode.
+** `fixOutdated` will check for outdated indexed document and reindex
+only those.
+
+This optional parameter must be passed as query parameter.
+
+It’s good to note as well that there is a limitation with the
+`fixOutdated` mode. As we first collect metadata of stored messages to
+compare them with the ones in the index, a failed `expunged` operation
+might not be well corrected (as the message might not exist anymore but
+still be indexed).
+
+Example:
+
+....
+curl -XPOST /messages?task=SolveInconsistencies&messagesPerSecond=200&mode=rebuildAll
+....
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+The scheduled task will have the following type
+`solve-message-inconsistencies` and the following
+`additionalInformation`:
+
+....
+{
+ "type":"solve-message-inconsistencies",
+ "timestamp":"2007-12-03T10:15:30Z",
+ "processedImapUidEntries": 2,
+ "processedMessageIdEntries": 1,
+ "addedMessageIdEntries": 1,
+ "updatedMessageIdEntries": 0,
+ "removedMessageIdEntries": 1,
+ "runningOptions":{
+ "messagesPerSecond": 200,
+ "mode":"REBUILD_ALL"
+ },
+ "fixedInconsistencies": [
+ {
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "d2bee791-7e63-11ea-883c-95b84008f979",
+ "uid": 1
+ },
+ {
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "d2bee792-7e63-11ea-883c-95b84008f979",
+ "uid": 2
+ }
+ ],
+ "errors": [
+ {
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "ffffffff-7e63-11ea-883c-95b84008f979",
+ "uid": 3
+ }
+ ]
+}
+....
+
+User actions concurrent to the inconsistency fixing task could result in
+concurrency issues. New inconsistencies could be created.
+
+However the source of truth will not be impacted, hence rerunning the
+task will eventually fix all issues.
+
+This task could be run safely online and can be scheduled on a recurring
+basis outside of peak traffic by an admin to ensure Cassandra message
+consistency.
+
+== Administrating user mailboxes
+
+=== Creating a mailbox
+
+....
+curl -XPUT http://ip:port/users/{usernameToBeUsed}/mailboxes/{mailboxNameToBeCreated}
+....
+
+Resource name `usernameToBeUsed` should be an existing user Resource
+name `mailboxNameToBeCreated` should not be empty, nor contain # & % *
+characters.
+
+Response codes:
+
+* 204: The mailbox now exists on the server
+* 400: Invalid mailbox name
+* 404: The user name does not exist
+
+To create nested mailboxes, for instance a work mailbox inside the INBOX
+mailbox, people should use the . separator. The sample query is:
+
+....
+curl -XDELETE http://ip:port/users/{usernameToBeUsed}/mailboxes/INBOX.work
+....
+
+=== Deleting a mailbox and its children
+
+....
+curl -XDELETE http://ip:port/users/{usernameToBeUsed}/mailboxes/{mailboxNameToBeDeleted}
+....
+
+Resource name `usernameToBeUsed` should be an existing user Resource
+name `mailboxNameToBeDeleted` should not be empty
+
+Response codes:
+
+* 204: The mailbox now does not exist on the server
+* 400: Invalid mailbox name
+* 404: The user name does not exist
+
+=== Testing existence of a mailbox
+
+....
+curl -XGET http://ip:port/users/{usernameToBeUsed}/mailboxes/{mailboxNameToBeTested}
+....
+
+Resource name `usernameToBeUsed` should be an existing user Resource
+name `mailboxNameToBeTested` should not be empty
+
+Response codes:
+
+* 204: The mailbox exists
+* 400: Invalid mailbox name
+* 404: The user name does not exist, the mailbox does not exist
+
+=== Listing user mailboxes
+
+....
+curl -XGET http://ip:port/users/{usernameToBeUsed}/mailboxes
+....
+
+The answer looks like:
+
+....
+[{"mailboxName":"INBOX"},{"mailboxName":"outbox"}]
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+Response codes:
+
+* 200: The mailboxes list was successfully retrieved
+* 404: The user name does not exist
+
+=== Deleting user mailboxes
+
+....
+curl -XDELETE http://ip:port/users/{usernameToBeUsed}/mailboxes
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+Response codes:
+
+* 204: The user do not have mailboxes anymore
+* 404: The user name does not exist
+
+=== Exporting user mailboxes
+
+....
+curl -XPOST http://ip:port/users/{usernameToBeUsed}/mailboxes?action=export
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned
+* 404: The user name does not exist
+
+The scheduled task will have the following type `MailboxesExportTask`
+and the following `additionalInformation`:
+
+....
+{
+ "type":"MailboxesExportTask",
+ "timestamp":"2007-12-03T10:15:30Z",
+ "username": "user",
+ "stage": "STARTING"
+}
+....
+
+=== ReIndexing a user mails
+
+....
+curl -XPOST http://ip:port/users/{usernameToBeUsed}/mailboxes?task=reIndex
+....
+
+Will schedule a task for reIndexing all the mails in ``user@domain.com''
+mailboxes (encoded above).
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed per
+second. Default is 50.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameter.
+
+An admin can also specify the reindexing mode it wants to use when
+running the task:
+
+* `mode` the reindexing mode used. There are 2 modes for the moment:
+** `rebuildAll` allows to rebuild all indexes. This is the default mode.
+** `fixOutdated` will check for outdated indexed document and reindex
+only those.
+
+This optional parameter must be passed as query parameter.
+
+It’s good to note as well that there is a limitation with the
+`fixOutdated` mode. As we first collect metadata of stored messages to
+compare them with the ones in the index, a failed `expunged` operation
+might not be well corrected (as the message might not exist anymore but
+still be indexed).
+
+Example:
+
+....
+curl -XPOST http://ip:port/users/{usernameToBeUsed}/mailboxes?task=reIndex&messagesPerSecond=200&mode=fixOutdated
+....
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+
+The scheduled task will have the following type `user-reindexing` and
+the following `additionalInformation`:
+
+....
+{
+ "type":"user-reindexing",
+ "runningOptions":{
+ "messagesPerSecond":200,
+ "mode":"FIX_OUTDATED"
+ },
+ "user":"user@domain.com",
+ "successfullyReprocessedMailCount":18,
+ "failedReprocessedMailCount": 3,
+ "mailboxFailures": ["12", "23" ],
+ "messageFailures": [
+ {
+ "mailboxId": "1",
+ "uids": [1, 36]
+ }]
+}
+....
+
+Warning: During the re-indexing, the result of search operations might
+be altered.
+
+Warning: Canceling this task should be considered unsafe as it will
+leave the currently reIndexed mailbox as partially indexed.
+
+Warning: While we have been trying to reduce the inconsistency window to
+a maximum (by keeping track of ongoing events), concurrent changes done
+during the reIndexing might be ignored.
+
+=== Recomputing User JMAP fast message view projection
+
+This action is only available for backends supporting JMAP protocol.
+
+Message fast view projection stores message properties expected to be
+fast to fetch but are actually expensive to compute, in order for
+GetMessages operation to be fast to execute for these properties.
+
+These projection items are asynchronously computed on mailbox events.
+
+You can force the full projection recomputation by calling the following
+endpoint:
+
+....
+curl -XPOST /users/{usernameToBeUsed}/mailboxes?task=recomputeFastViewProjectionItems
+....
+
+Will schedule a task for recomputing the fast message view projection
+for all mailboxes of `usernameToBeUsed`.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `messagesPerSecond` rate at which messages should be processed, per
+second. Defaults to 10.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameters.
+
+Example:
+
+....
+curl -XPOST /mailboxes?task=recomputeFastViewProjectionItems&messagesPerSecond=20
+....
+
+The scheduled task will have the following type
+`RecomputeUserFastViewProjectionItemsTask` and the following
+`additionalInformation`:
+
+....
+{
+ "type":"RecomputeUserFastViewProjectionItemsTask",
+ "username": "{usernameToBeUsed}",
+ "processedMessageCount": 3,
+ "failedMessageCount": 1,
+ "runningOptions": {
+ "messagesPerSecond":20
+ }
+}
+....
+
+Response codes:
+
+* 201: Success. Corresponding task id is returned.
+* 400: Error in the request. Details can be found in the reported error.
+* 404: User not found.
+
+== Administrating quotas by users
+
+=== Getting the quota for a user
+
+....
+curl -XGET http://ip:port/quota/users/{usernameToBeUsed}
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The answer is the details of the quota of that user.
+
+....
+{
+ "global": {
+ "count":252,
+ "size":242
+ },
+ "domain": {
+ "count":152,
+ "size":142
+ },
+ "user": {
+ "count":52,
+ "size":42
+ },
+ "computed": {
+ "count":52,
+ "size":42
+ },
+ "occupation": {
+ "size":13,
+ "count":21,
+ "ratio": {
+ "size":0.25,
+ "count":0.5,
+ "max":0.5
+ }
+ }
+}
+....
+
+* The `global` entry represent the quota limit allowed on this James
+server.
+* The `domain` entry represent the quota limit allowed for the user of
+that domain.
+* The `user` entry represent the quota limit allowed for this specific
+user.
+* The `computed` entry represent the quota limit applied for this user,
+resolved from the upper values.
+* The `occupation` entry represent the occupation of the quota for this
+user. This includes used count and size as well as occupation ratio
+(used / limit).
+
+Note that `quota` object can contain a fixed value, an empty value
+(null) or an unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 200: The user’s quota was successfully retrieved
+* 404: The user does not exist
+
+=== Updating the quota for a user
+
+....
+curl -XPUT http://ip:port/quota/users/{usernameToBeUsed}
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The body can contain a fixed value, an empty value (null) or an
+unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The user does not exist
+
+=== Getting the quota count for a user
+
+....
+curl -XGET http://ip:port/quota/users/{usernameToBeUsed}/count
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The user’s quota was successfully retrieved
+* 204: No quota count limit is defined at the user level for this user
+* 404: The user does not exist
+
+=== Updating the quota count for a user
+
+....
+curl -XPUT http://ip:port/quota/users/{usernameToBeUsed}/count
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The user does not exist
+
+=== Deleting the quota count for a user
+
+....
+curl -XDELETE http://ip:port/quota/users/{usernameToBeUsed}/count
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+* 404: The user does not exist
+
+=== Getting the quota size for a user
+
+....
+curl -XGET http://ip:port/quota/users/{usernameToBeUsed}/size
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The user’s quota was successfully retrieved
+* 204: No quota size limit is defined at the user level for this user
+* 404: The user does not exist
+
+=== Updating the quota size for a user
+
+....
+curl -XPUT http://ip:port/quota/users/{usernameToBeUsed}/size
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The user does not exist
+
+=== Deleting the quota size for a user
+
+....
+curl -XDELETE http://ip:port/quota/users/{usernameToBeUsed}/size
+....
+
+Resource name `usernameToBeUsed` should be an existing user
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+* 404: The user does not exist
+
+=== Searching user by quota ratio
+
+....
+curl -XGET 'http://ip:port/quota/users?minOccupationRatio=0.8&maxOccupationRatio=0.99&limit=100&offset=200&domain=domain.com'
+....
+
+Will return:
+
+....
+[
+ {
+ "username":"user@domain.com",
+ "detail": {
+ "global": {
+ "count":252,
+ "size":242
+ },
+ "domain": {
+ "count":152,
+ "size":142
+ },
+ "user": {
+ "count":52,
+ "size":42
+ },
+ "computed": {
+ "count":52,
+ "size":42
+ },
+ "occupation": {
+ "size":48,
+ "count":21,
+ "ratio": {
+ "size":0.9230,
+ "count":0.5,
+ "max":0.9230
+ }
+ }
+ }
+ },
+ ...
+]
+....
+
+Where:
+
+* *minOccupationRatio* is a query parameter determining the minimum
+occupation ratio of users to be returned.
+* *maxOccupationRatio* is a query parameter determining the maximum
+occupation ratio of users to be returned.
+* *domain* is a query parameter determining the domain of users to be
+returned.
+* *limit* is a query parameter determining the maximum number of users
+to be returned.
+* *offset* is a query parameter determining the number of users to skip.
+
+Please note that users are alphabetically ordered on username.
+
+The response is a list of usernames, with attached quota details as
+defined link:#getting-the-quota-for-a-user[here].
+
+Response codes:
+
+* 200: List of users had successfully been returned.
+* 400: Validation issues with parameters
+
+=== Recomputing current quotas for users
+
+....
+curl -XPOST /quota/users?task=RecomputeCurrentQuotas
+....
+
+Will recompute current quotas (count and size) for all users stored in
+James.
+
+James maintains per quota a projection for current quota count and size.
+As with any projection, it can go out of sync, leading to inconsistent
+results being returned to the client.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+An admin can specify the concurrency that should be used when running
+the task:
+
+* `usersPerSecond` rate at which users quotas should be reprocessed, per
+second. Defaults to 1.
+
+This optional parameter must have a strictly positive integer as a value
+and be passed as query parameters.
+
+Example:
+
+....
+curl -XPOST /quota/users?task=RecomputeCurrentQuotas&usersPerSecond=20
+....
+
+The scheduled task will have the following type
+`recompute-current-quotas` and the following `additionalInformation`:
+
+....
+{
+ "type":"recompute-current-quotas",
+ "processedQuotaRoots": 3,
+ "failedQuotaRoots": ["#private&bob@localhost"],
+ "runningOptions": {
+ "usersPerSecond":20
+ }
+}
+....
+
+*WARNING*: this task do not take into account concurrent modifications
+upon a single current quota re-computation. Rerunning the task will
+_eventually_ provide the consistent result.
+
+== Administrating quotas by domains
+
+=== Getting the quota for a domain
+
+....
+curl -XGET http://ip:port/quota/domains/{domainToBeUsed}
+....
+
+Resource name `domainToBeUsed` should be an existing domain. For
+example:
+
+....
+curl -XGET http://ip:port/quota/domains/james.org
+....
+
+The answer will detail the default quota applied to users belonging to
+that domain:
+
+....
+{
+ "global": {
+ "count":252,
+ "size":null
+ },
+ "domain": {
+ "count":null,
+ "size":142
+ },
+ "computed": {
+ "count":252,
+ "size":142
+ }
+}
+....
+
+* The `global` entry represents the quota limit defined on this James
+server by default.
+* The `domain` entry represents the quota limit allowed for the user of
+that domain by default.
+* The `computed` entry represents the quota limit applied for the users
+of that domain, by default, resolved from the upper values.
+
+Note that `quota` object can contain a fixed value, an empty value
+(null) or an unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 200: The domain’s quota was successfully retrieved
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Updating the quota for a domain
+
+....
+curl -XPUT http://ip:port/quota/domains/{domainToBeUsed}
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+The body can contain a fixed value, an empty value (null) or an
+unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Getting the quota count for a domain
+
+....
+curl -XGET http://ip:port/quota/domains/{domainToBeUsed}/count
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The domain’s quota was successfully retrieved
+* 204: No quota count limit is defined at the domain level for this
+domain
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Updating the quota count for a domain
+
+....
+curl -XPUT http://ip:port/quota/domains/{domainToBeUsed}/count
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Deleting the quota count for a domain
+
+....
+curl -XDELETE http://ip:port/quota/domains/{domainToBeUsed}/count
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Getting the quota size for a domain
+
+....
+curl -XGET http://ip:port/quota/domains/{domainToBeUsed}/size
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The domain’s quota was successfully retrieved
+* 204: No quota size limit is defined at the domain level for this
+domain
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Updating the quota size for a domain
+
+....
+curl -XPUT http://ip:port/quota/domains/{domainToBeUsed}/size
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+* 404: The domain does not exist
+* 405: Domain Quota configuration not supported when virtual hosting is
+desactivated.
+
+=== Deleting the quota size for a domain
+
+....
+curl -XDELETE http://ip:port/quota/domains/{domainToBeUsed}/size
+....
+
+Resource name `domainToBeUsed` should be an existing domain.
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+* 404: The domain does not exist
+
+== Administrating global quotas
+
+=== Getting the global quota
+
+....
+curl -XGET http://ip:port/quota
+....
+
+The answer is the details of the global quota.
+
+....
+{
+ "count":252,
+ "size":242
+}
+....
+
+Note that `quota` object can contain a fixed value, an empty value
+(null) or an unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 200: The quota was successfully retrieved
+
+=== Updating global quota
+
+....
+curl -XPUT http://ip:port/quota
+....
+
+The body can contain a fixed value, an empty value (null) or an
+unlimited value (-1):
+
+....
+{"count":52,"size":42}
+
+{"count":null,"size":null}
+
+{"count":52,"size":-1}
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+
+=== Getting the global quota count
+
+....
+curl -XGET http://ip:port/quota/count
+....
+
+Resource name usernameToBeUsed should be an existing user
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The quota was successfully retrieved
+* 204: No quota count limit is defined at the global level
+
+=== Updating the global quota count
+
+....
+curl -XPUT http://ip:port/quota/count
+....
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+
+=== Deleting the global quota count
+
+....
+curl -XDELETE http://ip:port/quota/count
+....
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+
+=== Getting the global quota size
+
+....
+curl -XGET http://ip:port/quota/size
+....
+
+The answer looks like:
+
+....
+52
+....
+
+Response codes:
+
+* 200: The quota was successfully retrieved
+* 204: No quota size limit is defined at the global level
+
+=== Updating the global quota size
+
+....
+curl -XPUT http://ip:port/quota/size
+....
+
+The body can contain a fixed value or an unlimited value (-1):
+
+....
+52
+....
+
+Response codes:
+
+* 204: The quota has been updated
+* 400: The body is not a positive integer neither an unlimited value
+(-1).
+
+=== Deleting the global quota size
+
+....
+curl -XDELETE http://ip:port/quota/size
+....
+
+Response codes:
+
+* 204: The quota has been updated to unlimited value.
+
+== Cassandra Schema upgrades
+
+Cassandra upgrades implies the creation of a new table. Thus restarting
+James is needed, as new tables are created on restart.
+
+Once done, we ship code that tries to read from new tables, and if not
+possible backs up to old tables. You can thus safely run without running
+additional migrations.
+
+On the fly migration can be enabled. However, one might want to force
+the migration in a controlled fashion, and update automatically current
+schema version used (assess in the database old versions is no more
+used, as the corresponding tables are empty). Note that this process is
+safe: we ensure the service is not running concurrently on this James
+instance, that it does not bump version upon partial failures, that race
+condition in version upgrades will be idempotent, etc…
+
+These schema updates can be triggered by webadmin using the Cassandra
+backend.
+
+Note that currently the progress can be tracked by logs.
+
+* link:#Retrieving_current_Cassandra_schema_version[Retrieving current
+Cassandra schema version]
+* link:#Retrieving_latest_available_Cassandra_schema_version[Retrieving
+latest available Cassandra schema version]
+* link:#Upgrading_to_a_specific_version[Upgrading to a specific version]
+* link:#Upgrading_to_the_latest_version[Upgrading to the latest version]
+
+=== Retrieving current Cassandra schema version
+
+....
+curl -XGET http://ip:port/cassandra/version
+....
+
+Will return:
+
+....
+{"version": 2}
+....
+
+Where the number corresponds to the current schema version of the
+database you are using.
+
+Response codes:
+
+* 200: Success
+
+=== Retrieving latest available Cassandra schema version
+
+....
+curl -XGET http://ip:port/cassandra/version/latest
+....
+
+Will return:
+
+....
+{"version": 3}
+....
+
+Where the number corresponds to the latest available schema version of
+the database you are using. This means you can be migrating to this
+schema version.
+
+Response codes:
+
+* 200: Success
+
+=== Upgrading to a specific version
+
+....
+curl -XPOST http://ip:port/cassandra/version/upgrade -d '3'
+....
+
+Will schedule the run of the migrations you need to reach schema version
+3.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 200: Success. The scheduled task `taskId` is returned.
+* 400: The version is invalid. The version should be a strictly positive
+number.
+* 410: Error while planning this migration. This resource is gone away.
+Reason is mentionned in the body.
+
+Note that several calls to this endpoint will be run in a sequential
+pattern.
+
+If the server restarts during the migration, the migration is silently
+aborted.
+
+The scheduled task will have the following type `cassandra-migration`
+and the following `additionalInformation`:
+
+....
+{"targetVersion":3}
+....
+
+=== Upgrading to the latest version
+
+....
+curl -XPOST http://ip:port/cassandra/version/upgrade/latest
+....
+
+Will schedule the run of the migrations you need to reach the latest
+schema version.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 200: Success. The scheduled task `taskId` is returned.
+* 410: Error while planning this migration. This resource is gone away.
+Reason is mentionned in the body.
+
+Note that several calls to this endpoint will be run in a sequential
+pattern.
+
+If the server restarts during the migration, the migration is silently
+aborted.
+
+The scheduled task will have the following type `cassandra-migration`
+and the following `additionalInformation`:
+
+....
+{"toVersion":2}
+....
+
+== Correcting ghost mailbox
+
+This is a temporary workaround for the *Ghost mailbox* bug encountered
+using the Cassandra backend, as described in MAILBOX-322.
+
+You can use the mailbox merging feature in order to merge the old
+``ghosted'' mailbox with the new one.
+
+....
+curl -XPOST http://ip:port/cassandra/mailbox/merging \
+ -d '{"mergeOrigin":"{id1}", "mergeDestination":"{id2}"}' \
+ -H "Content-Type: application/json"
+....
+
+Will scedule a task for :
+
+* Delete references to `id1` mailbox
+* Move it’s messages into `id2` mailbox
+* Union the rights of both mailboxes
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 400: Unable to parse the body.
+
+The scheduled task will have the following type `mailbox-merging` and
+the following `additionalInformation`:
+
+....
+{
+ "oldMailboxId":"5641376-02ed-47bd-bcc7-76ff6262d92a",
+ "newMailboxId":"4555159-52ae-895f-ccb7-586a4412fb50",
+ "totalMessageCount": 1,
+ "messageMovedCount": 1,
+ "messageFailedCount": 0
+}
+....
+
+== Creating address group
+
+You can use *webadmin* to define address groups.
+
+When a specific email is sent to the group mail address, every group
+member will receive it.
+
+Note that the group mail address is virtual: it does not correspond to
+an existing user.
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+mailet] to be configured.
+
+Note that email addresses are restricted to ASCII character set. Mail
+addresses not matching this criteria will be rejected.
+
+=== Listing groups
+
+....
+curl -XGET http://ip:port/address/groups
+....
+
+Will return the groups as a list of JSON Strings representing mail
+addresses. For instance:
+
+....
+["group1@domain.com", "group2@domain.com"]
+....
+
+Response codes:
+
+* 200: Success
+
+=== Listing members of a group
+
+....
+curl -XGET http://ip:port/address/groups/group@domain.com
+....
+
+Will return the group members as a list of JSON Strings representing
+mail addresses. For instance:
+
+....
+["member1@domain.com", "member2@domain.com"]
+....
+
+Response codes:
+
+* 200: Success
+* 400: Group structure is not valid
+* 404: The group does not exist
+
+=== Adding a group member
+
+....
+curl -XPUT http://ip:port/address/groups/group@domain.com/member@domain.com
+....
+
+Will add member@domain.com to group@domain.com, creating the group if
+needed
+
+Response codes:
+
+* 204: Success
+* 400: Group structure or member is not valid
+* 400: Domain in the source is not managed by the DomainList
+* 409: Requested group address is already used for another purpose
+
+=== Removing a group member
+
+....
+curl -XDELETE http://ip:port/address/groups/group@domain.com/member@domain.com
+....
+
+Will remove member@domain.com from group@domain.com, removing the group
+if group is empty after deletion
+
+Response codes:
+
+* 204: Success
+* 400: Group structure or member is not valid
+
+== Creating address forwards
+
+You can use *webadmin* to define address forwards.
+
+When a specific email is sent to the base mail address, every forward
+destination addresses will receive it.
+
+Please note that the base address can be optionaly part of the forward
+destination. In that case, the base recipient also receive a copy of the
+mail. Otherwise he is ommitted.
+
+Forwards can be defined for existing users. It then defers from
+``groups''.
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+mailet] to be configured.
+
+Note that email addresses are restricted to ASCII character set. Mail
+addresses not matching this criteria will be rejected.
+
+=== Listing Forwards
+
+....
+curl -XGET http://ip:port/address/forwards
+....
+
+Will return the users having forwards configured as a list of JSON
+Strings representing mail addresses. For instance:
+
+....
+["user1@domain.com", "user2@domain.com"]
+....
+
+Response codes:
+
+* 200: Success
+
+=== Listing destinations in a forward
+
+....
+curl -XGET http://ip:port/address/forwards/user@domain.com
+....
+
+Will return the destination addresses of this forward as a list of JSON
+Strings representing mail addresses. For instance:
+
+....
+[
+ {"mailAddress":"destination1@domain.com"},
+ {"mailAddress":"destination2@domain.com"}
+]
+....
+
+Response codes:
+
+* 200: Success
+* 400: Forward structure is not valid
+* 404: The given user don’t have forwards or does not exist
+
+=== Adding a new destination to a forward
+
+....
+curl -XPUT http://ip:port/address/forwards/user@domain.com/targets/destination@domain.com
+....
+
+Will add destination@domain.com to user@domain.com, creating the forward
+if needed
+
+Response codes:
+
+* 204: Success
+* 400: Forward structure or member is not valid
+* 400: Domain in the source is not managed by the DomainList
+* 404: Requested forward address does not match an existing user
+
+=== Removing a destination of a forward
+
+....
+curl -XDELETE http://ip:port/address/forwards/user@domain.com/targets/destination@domain.com
+....
+
+Will remove destination@domain.com from user@domain.com, removing the
+forward if forward is empty after deletion
+
+Response codes:
+
+* 204: Success
+* 400: Forward structure or member is not valid
+
+== Creating address aliases
+
+You can use *webadmin* to define aliases for an user.
+
+When a specific email is sent to the alias address, the destination
+address of the alias will receive it.
+
+Aliases can be defined for existing users.
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+mailet] to be configured.
+
+Note that email addresses are restricted to ASCII character set. Mail
+addresses not matching this criteria will be rejected.
+
+=== Listing users with aliases
+
+....
+curl -XGET http://ip:port/address/aliases
+....
+
+Will return the users having aliases configured as a list of JSON
+Strings representing mail addresses. For instance:
+
+....
+["user1@domain.com", "user2@domain.com"]
+....
+
+Response codes:
+
+* 200: Success
+
+=== Listing alias sources of an user
+
+....
+curl -XGET http://ip:port/address/aliases/user@domain.com
+....
+
+Will return the aliases of this user as a list of JSON Strings
+representing mail addresses. For instance:
+
+....
+[
+ {"source":"alias1@domain.com"},
+ {"source":"alias2@domain.com"}
+]
+....
+
+Response codes:
+
+* 200: Success
+* 400: Alias structure is not valid
+
+=== Adding a new alias to an user
+
+....
+curl -XPUT http://ip:port/address/aliases/user@domain.com/sources/alias@domain.com
+....
+
+Will add alias@domain.com to user@domain.com, creating the alias if
+needed
+
+Response codes:
+
+* 204: OK
+* 400: Alias structure or member is not valid
+* 400: The alias source exists as an user already
+* 400: Source and destination can’t be the same!
+* 400: Domain in the destination or source is not managed by the
+DomainList
+
+=== Removing an alias of an user
+
+....
+curl -XDELETE http://ip:port/address/aliases/user@domain.com/sources/alias@domain.com
+....
+
+Will remove alias@domain.com from user@domain.com, removing the alias if
+needed
+
+Response codes:
+
+* 204: OK
+* 400: Alias structure or member is not valid
+
+== Creating domain mappings
+
+You can use *webadmin* to define domain mappings.
+
+Given a configured source (from) domain and a destination (to) domain,
+when an email is sent to an address belonging to the source domain, then
+the domain part of this address is overwritten, the destination domain
+is then used. A source (from) domain can have many destination (to)
+domains.
+
+For example: with a source domain `james.apache.org` maps to two
+destination domains `james.org` and `apache-james.org`, when a mail is
+sent to `admin@james.apache.org`, then it will be routed to
+`admin@james.org` and `admin@apache-james.org`
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+mailet] to be configured.
+
+Note that email addresses are restricted to ASCII character set. Mail
+addresses not matching this criteria will be rejected.
+
+=== Listing all domain mappings
+
+....
+curl -XGET http://ip:port/domainMappings
+....
+
+Will return all configured domain mappings
+
+....
+{
+ "firstSource.org" : ["firstDestination.com", "secondDestination.net"],
+ "secondSource.com" : ["thirdDestination.com", "fourthDestination.net"],
+}
+....
+
+Response codes:
+
+* 200: OK
+
+=== Listing all destination domains for a source domain
+
+....
+curl -XGET http://ip:port/domainMappings/sourceDomain.tld
+....
+
+With `sourceDomain.tld` as the value passed to `fromDomain` resource
+name, the API will return all destination domains configured to that
+domain
+
+....
+["firstDestination.com", "secondDestination.com"]
+....
+
+Response codes:
+
+* 200: OK
+* 400: The `fromDomain` resource name is invalid
+* 404: The `fromDomain` resource name is not found
+
+=== Adding a domain mapping
+
+....
+curl -XPUT http://ip:port/domainMappings/sourceDomain.tld
+....
+
+Body:
+
+....
+destination.tld
+....
+
+With `sourceDomain.tld` as the value passed to `fromDomain` resource
+name, the API will add a destination domain specified in the body to
+that domain
+
+Response codes:
+
+* 204: OK
+* 400: The `fromDomain` resource name is invalid
+* 400: The destination domain specified in the body is invalid
+
+=== Removing a domain mapping
+
+....
+curl -XDELETE http://ip:port/domainMappings/sourceDomain.tld
+....
+
+Body:
+
+....
+destination.tld
+....
+
+With `sourceDomain.tld` as the value passed to `fromDomain` resource
+name, the API will remove a destination domain specified in the body
+mapped to that domain
+
+Response codes:
+
+* 204: OK
+* 400: The `fromDomain` resource name is invalid
+* 400: The destination domain specified in the body is invalid
+
+== Creating regex mapping
+
+You can use *webadmin* to create regex mappings.
+
+A regex mapping contains a mapping source and a Java Regular Expression
+(regex) in String as the mapping value. Everytime, if a mail containing
+a recipient matched with the mapping source, then that mail will be
+re-routed to a new recipient address which is re written by the regex.
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+API] to be configured.
+
+=== Adding a regex mapping
+
+....
+POST /mappings/regex/mappingSource/targets/regex
+....
+
+Where:
+
+* the `mappingSource` is the path parameter represents for the Regex
+Mapping mapping source
+* the `regex` is the path parameter represents for the Regex Mapping
+regex
+
+The route will add a regex mapping made from `mappingSource` and `regex`
+to RecipientRewriteTable.
+
+Example:
+
+....
+curl -XPOST http://ip:port/mappings/regex/james@domain.tld/targets/james@.*:james-intern@james.org
+....
+
+Response codes:
+
+* 204: Mapping added successfully.
+* 400: Invalid `mappingSource` path parameter.
+* 400: Invalid `regex` path parameter.
+
+=== Removing a regex mapping
+
+....
+DELETE /mappings/regex/{mappingSource}/targets/{regex}
+....
+
+Where:
+
+* the `mappingSource` is the path parameter representing the Regex
+Mapping mapping source
+* the `regex` is the path parameter representing the Regex Mapping regex
+
+The route will remove the regex mapping made from `regex` from the
+mapping source `mappingSource` to RecipientRewriteTable.
+
+Example:
+
+....
+curl -XDELETE http://ip:port/mappings/regex/james@domain.tld/targets/[O_O]:james-intern@james.org
+....
+
+Response codes:
+
+* 204: Mapping deleted successfully.
+* 400: Invalid `mappingSource` path parameter.
+* 400: Invalid `regex` path parameter.
+
+== Address Mappings
+
+You can use *webadmin* to define address mappings.
+
+When a specific email is sent to the base mail address, every
+destination addresses will receive it.
+
+This feature uses (TODO)
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+and requires the
+https://github.com/apache/james-project/blob/master/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/RecipientRewriteTable.java[RecipientRewriteTable
+mailet] to be configured.
+
+Note that email addresses are restricted to ASCII character set. Mail
+addresses not matching this criteria will be rejected.
+
+Please use address mappings with caution, as it’s not a typed address.
+If you know the type of your address (forward, alias, domain, group,
+etc), prefer using the corresponding routes to those types.
+
+Here are the following actions available on address mappings:
+
+=== Add an address mapping
+
+....
+curl -XPOST http://ip:port/mappings/address/{mappingSource}/targets/{destinationAddress}
+....
+
+Add an address mapping to the Recipients rewrite table
+Mapping source is the value of \{mappingSource} Mapping destination is
+the value of \{destinationAddress} Type of mapping destination is
+Address
+
+Response codes:
+
+* 204: Action successfully performed
+* 400: Invalid parameters
+
+=== Remove an address mapping
+
+....
+curl -XDELETE http://ip:port/mappings/address/{mappingSource}/targets/{destinationAddress}
+....
+
+* Remove an address mapping from the Recipients rewrite table
+* Mapping source is the value of `mappingSource`
+* Mapping destination is the value of `destinationAddress`
+* Type of mapping destination is Address
+
+Response codes:
+
+* 204: Action successfully performed
+* 400: Invalid parameters
+
+== List all mappings
+
+....
+curl -XGET http://ip:port/mappings
+....
+
+Get all mappings from the
+link:/server/config-recipientrewritetable.html[Recipients rewrite table]
+Supported mapping types are the following:
+
+(TODO)
+
+* link:#Creating_address_aliases[Alias]
+* link:#Address_Mappings[Address]
+* link:#Creating_address_domain[Domain]
+* Error
+* link:#Creating_address_forwards[Forward]
+* link:#Creating_address_group[Group]
+* Regex
+
+Response body:
+
+....
+{
+ "alias@domain.tld": [
+ {
+ "type": "Alias",
+ "mapping": "user@domain.tld"
+ },
+ {
+ "type": "Group",
+ "mapping": "group-user@domain.tld"
+ }
+ ],
+ "aliasdomain.tld": [
+ {
+ "type": "Domain",
+ "mapping": "realdomain.tld"
+ }
+ ],
+ "group@domain.tld": [
+ {
+ "type": "Address",
+ "mapping": "user@domain.tld"
+ }
+ ]
+}
+....
+
+Response code:
+
+* 200: OK
+
+== User Mappings
+
+=== Listing User Mappings
+
+This endpoint allows receiving all mappings of a corresponding user.
+
+....
+curl -XGET http://ip:port/mappings/user/{userAddress}
+....
+
+Return all mappings of a user where:
+
+* `userAddress`: is the selected user
+
+Response body:
+
+....
+[
+ {
+ "type": "Address",
+ "mapping": "user123@domain.tld"
+ },
+ {
+ "type": "Alias",
+ "mapping": "aliasuser123@domain.tld"
+ },
+ {
+ "type": "Group",
+ "mapping": "group123@domain.tld"
+ }
+]
+....
+
+Response codes:
+
+* 200: OK
+* 400: Invalid parameter value
+
+== Administrating mail repositories
+
+=== Create a mail repository
+
+....
+curl -XPUT http://ip:port/mailRepositories/{encodedPathOfTheRepository}?protocol={someProtocol}
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of the created mail repository. Example:
+
+....
+curl -XPUT http://ip:port/mailRepositories/mailRepo?protocol=file
+....
+
+Response codes:
+
+* 204: The repository is created
+
+=== Listing mail repositories
+
+....
+curl -XGET http://ip:port/mailRepositories
+....
+
+The answer looks like:
+
+....
+[
+ {
+ "repository": "var/mail/error/",
+ "path": "var%2Fmail%2Ferror%2F"
+ },
+ {
+ "repository": "var/mail/relay-denied/",
+ "path": "var%2Fmail%2Frelay-denied%2F"
+ },
+ {
+ "repository": "var/mail/spam/",
+ "path": "var%2Fmail%2Fspam%2F"
+ },
+ {
+ "repository": "var/mail/address-error/",
+ "path": "var%2Fmail%2Faddress-error%2F"
+ }
+]
+....
+
+You can use `id`, the encoded URL of the repository, to access it in
+later requests.
+
+Response codes:
+
+* 200: The list of mail repositories
+
+=== Getting additional information for a mail repository
+
+....
+curl -XGET http://ip:port/mailRepositories/{encodedPathOfTheRepository}
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Example:
+
+....
+curl -XGET http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F
+....
+
+The answer looks like:
+
+....
+{
+ "repository": "var/mail/error/",
+ "path": "mail%2Ferror%2F",
+ "size": 243
+}
+....
+
+Response codes:
+
+* 200: Additonnal information for that repository
+* 404: This repository can not be found
+
+=== Listing mails contained in a mail repository
+
+....
+curl -XGET http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Example:
+
+....
+curl -XGET http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails
+....
+
+The answer will contains all mailKey contained in that repository.
+
+....
+[
+ "mail-key-1",
+ "mail-key-2",
+ "mail-key-3"
+]
+....
+
+Note that this can be used to read mail details.
+
+You can pass additional URL parameters to this call in order to limit
+the output: - A limit: no more elements than the specified limit will be
+returned. This needs to be strictly positive. If no value is specified,
+no limit will be applied. - An offset: allow to skip elements. This
+needs to be positive. Default value is zero.
+
+Example:
+
+....
+curl -XGET 'http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails?limit=100&offset=500'
+....
+
+Response codes:
+
+* 200: The list of mail keys contained in that mail repository
+* 400: Invalid parameters
+* 404: This repository can not be found
+
+=== Reading/downloading a mail details
+
+....
+curl -XGET http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails/mailKey
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Resource name `mailKey` should be the
+key of a mail stored in that repository. Example:
+
+....
+curl -XGET http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails/mail-key-1
+....
+
+If the Accept header in the request is ``application/json'', then the
+response looks like:
+
+....
+{
+ "name": "mail-key-1",
+ "sender": "sender@domain.com",
+ "recipients": ["recipient1@domain.com", "recipient2@domain.com"],
+ "state": "address-error",
+ "error": "A small message explaining what happened to that mail...",
+ "remoteHost": "111.222.333.444",
+ "remoteAddr": "127.0.0.1",
+ "lastUpdated": null
+}
+....
+
+If the Accept header in the request is ``message/rfc822'', then the
+response will be the _eml_ file itself.
+
+Additional query parameter `additionalFields` add the existing
+information to the response for the supported values (only work with
+``application/json'' Accept header):
+
+* attributes
+* headers
+* textBody
+* htmlBody
+* messageSize
+* perRecipientsHeaders
+
+....
+curl -XGET http://ip:port/mailRepositories/file%3A%2F%2Fvar%2Fmail%2Ferror%2F/mails/mail-key-1?additionalFields=attributes,headers,textBody,htmlBody,messageSize,perRecipientsHeaders
+....
+
+Give the following kind of response:
+
+....
+{
+ "name": "mail-key-1",
+ "sender": "sender@domain.com",
+ "recipients": ["recipient1@domain.com", "recipient2@domain.com"],
+ "state": "address-error",
+ "error": "A small message explaining what happened to that mail...",
+ "remoteHost": "111.222.333.444",
+ "remoteAddr": "127.0.0.1",
+ "lastUpdated": null,
+ "attributes": {
+ "name2": "value2",
+ "name1": "value1"
+ },
+ "perRecipientsHeaders": {
+ "third@party": {
+ "headerName1": [
+ "value1",
+ "value2"
+ ],
+ "headerName2": [
+ "value3",
+ "value4"
+ ]
+ }
+ },
+ "headers": {
+ "headerName4": [
+ "value6",
+ "value7"
+ ],
+ "headerName3": [
+ "value5",
+ "value8"
+ ]
+ },
+ "textBody": "My body!!",
+ "htmlBody": "My <em>body</em>!!",
+ "messageSize": 42424242
+}
+....
+
+Response codes:
+
+* 200: Details of the mail
+* 404: This repository or mail can not be found
+
+=== Removing a mail from a mail repository
+
+....
+curl -XDELETE http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails/mailKey
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Resource name `mailKey` should be the
+key of a mail stored in that repository. Example:
+
+....
+curl -XDELETE http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails/mail-key-1
+....
+
+Response codes:
+
+* 204: This mail no longer exists in this repository
+* 404: This repository can not be found
+
+=== Removing all mails from a mail repository
+
+....
+curl -XDELETE http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Example:
+
+....
+curl -XDELETE http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 404: Could not find that mail repository
+
+The scheduled task will have the following type `clear-mail-repository`
+and the following `additionalInformation`:
+
+....
+{
+ "mailRepositoryPath":"var/mail/error/",
+ "initialCount": 243,
+ "remainingCount": 17
+}
+....
+
+=== Reprocessing mails from a mail repository
+
+Sometime, you want to re-process emails stored in a mail repository. For
+instance, you can make a configuration error, or there can be a James
+bug that makes processing of some mails fail. Those mail will be stored
+in a mail repository. Once you solved the problem, you can reprocess
+them.
+
+To reprocess mails from a repository:
+
+....
+curl -XPATCH http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails?action=reprocess
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource path
+of an existing mail repository. Example:
+
+For instance:
+
+....
+curl -XPATCH http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails?action=reprocess
+....
+
+Additional query parameters are supported: - `queue` allows you to
+target the mail queue you want to enqueue the mails in. Defaults to
+`spool`. - `processor` allows you to overwrite the state of the
+reprocessing mails, and thus select the processors they will start their
+processing in. Defaults to the `state` field of each processed email.
+
+For instance:
+
+....
+curl -XPATCH 'http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails?action=reprocess&processor=transport&queue=spool'
+....
+
+Note that the `action` query parameter is compulsary and can only take
+value `reprocess`.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 404: Could not find that mail repository
+
+The scheduled task will have the following type `reprocessing-all` and
+the following `additionalInformation`:
+
+....
+{
+ "mailRepositoryPath":"var/mail/error/",
+ "targetQueue":"spool",
+ "targetProcessor":"transport",
+ "initialCount": 243,
+ "remainingCount": 17
+}
+....
+
+=== Reprocessing a specific mail from a mail repository
+
+To reprocess a specific mail from a mail repository:
+
+....
+curl -XPATCH http://ip:port/mailRepositories/{encodedPathOfTheRepository}/mails/mailKey?action=reprocess
+....
+
+Resource name `encodedPathOfTheRepository` should be the resource id of
+an existing mail repository. Resource name `mailKey` should be the key
+of a mail stored in that repository. Example:
+
+For instance:
+
+....
+curl -XPATCH http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails/name1?action=reprocess
+....
+
+Additional query parameters are supported: - `queue` allows you to
+target the mail queue you want to enqueue the mails in. Defaults to
+`spool`. - `processor` allows you to overwrite the state of the
+reprocessing mails, and thus select the processors they will start their
+processing in. Defaults to the `state` field of each processed email.
+
+While `processor` being an optional parameter, not specifying it will
+result reprocessing the mails in their current state
+(https://james.apache.org/server/feature-mailetcontainer.html#Processors[see
+documentation about processors and state]). Consequently, only few cases
+will give a different result, definitively storing them out of the mail
+repository.
+
+For instance:
+
+....
+curl -XPATCH 'http://ip:port/mailRepositories/var%2Fmail%2Ferror%2F/mails/name1?action=reprocess&processor=transport&queue=spool'
+....
+
+Note that the `action` query parameter is compulsary and can only take
+value `reprocess`.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 404: Could not find that mail repository
+
+The scheduled task will have the following type `reprocessing-one` and
+the following `additionalInformation`:
+
+....
+{
+ "mailRepositoryPath":"var/mail/error/",
+ "targetQueue":"spool",
+ "targetProcessor":"transport",
+ "mailKey":"name1"
+}
+....
+
+== Administrating mail queues
+
+=== Listing mail queues
+
+....
+curl -XGET http://ip:port/mailQueues
+....
+
+The answer looks like:
+
+....
+["outgoing","spool"]
+....
+
+Response codes:
+
+* 200: The list of mail queues
+
+=== Getting a mail queue details
+
+....
+curl -XGET http://ip:port/mailQueues/{mailQueueName}
+....
+
+Resource name `mailQueueName` is the name of a mail queue, this command
+will return the details of the given mail queue. For instance:
+
+....
+{"name":"outgoing","size":0}
+....
+
+Response codes:
+
+* 200: Success
+* 400: Mail queue is not valid
+* 404: The mail queue does not exist
+
+=== Listing the mails of a mail queue
+
+....
+curl -XGET http://ip:port/mailQueues/{mailQueueName}/mails
+....
+
+Additional URL query parameters:
+
+* `limit`: Maximum number of mails returned in a single call. Only
+strictly positive integer values are accepted. Example:
+
+....
+curl -XGET http://ip:port/mailQueues/{mailQueueName}/mails?limit=100
+....
+
+The answer looks like:
+
+....
+[{
+ "name": "Mail1516976156284-8b3093b9-eebf-4c40-9c26-1450f4fcdc3c-to-test.com",
+ "sender": "user@james.linagora.com",
+ "recipients": ["someone@test.com"],
+ "nextDelivery": "1969-12-31T23:59:59.999Z"
+}]
+....
+
+Response codes:
+
+* 200: Success
+* 400: Mail queue is not valid or limit is invalid
+* 404: The mail queue does not exist
+
+=== Deleting mails from a mail queue
+
+....
+curl -XDELETE http://ip:port/mailQueues/{mailQueueName}/mails?sender=senderMailAddress
+....
+
+This request should have exactly one query parameter from the following
+list:
+
+* sender: which is a mail address (i.e. sender@james.org)
+* name: which is a string
+* recipient: which is a mail address (i.e. recipient@james.org)
+
+The mails from the given mail queue matching the query parameter will be
+deleted.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 400: Invalid request
+* 404: The mail queue does not exist
+
+The scheduled task will have the following type
+`delete-mails-from-mail-queue` and the following
+`additionalInformation`:
+
+....
+{
+ "queue":"outgoing",
+ "initialCount":10,
+ "remainingCount": 5,
+ "sender": "sender@james.org",
+ "name": "Java Developer",
+ "recipient: "recipient@james.org"
+}
+....
+
+=== Clearing a mail queue
+
+....
+curl -XDELETE http://ip:port/mailQueues/{mailQueueName}/mails
+....
+
+All mails from the given mail queue will be deleted.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* 400: Invalid request
+* 404: The mail queue does not exist
+
+The scheduled task will have the following type `clear-mail-queue` and
+the following `additionalInformation`:
+
+....
+{
+ "queue":"outgoing",
+ "initialCount":10,
+ "remainingCount": 0
+}
+....
+
+=== Flushing mails from a mail queue
+
+....
+curl -XPATCH http://ip:port/mailQueues/{mailQueueName}?delayed=true \
+ -d '{"delayed": false}' \
+ -H "Content-Type: application/json"
+....
+
+This request should have the query parameter _delayed_ set to _true_, in
+order to indicate only delayed mails are affected. The payload should
+set the `delayed` field to false inorder to remove the delay. This is
+the only supported combination, and it performs a flush.
+
+The mails delayed in the given mail queue will be flushed.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 204: Success (No content)
+* 400: Invalid request
+* 404: The mail queue does not exist
+
+== Administrating DLP Configuration
+
+DLP (stands for Data Leak Prevention) is supported by James. A DLP
+matcher will, on incoming emails, execute regular expressions on email
+sender, recipients or content, in order to report suspicious emails to
+an administrator. WebAdmin can be used to manage these DLP rules on a
+per `senderDomain` basis.
+
+`senderDomain` is domain of the sender of incoming emails, for example:
+`apache.org`, `james.org`,… Each `senderDomain` correspond to a distinct
+DLP configuration.
+
+=== List DLP configuration by sender domain
+
+Retrieve a DLP configuration for corresponding `senderDomain`, a
+configuration contains list of configuration items
+
+....
+curl -XGET http://ip:port/dlp/rules/{senderDomain}
+....
+
+Response codes:
+
+* 200: A list of dlp configuration items is returned
+* 400: Invalid `senderDomain` or payload in request
+* 404: The domain does not exist.
+
+This is an example of returned body. The rules field is a list of rules
+as described below.
+
+....
+{"rules : [
+ {
+ "id": "1",
+ "expression": "james.org",
+ "explanation": "Find senders or recipients containing james[any char]org",
+ "targetsSender": true,
+ "targetsRecipients": true,
+ "targetsContent": false
+ },
+ {
+ "id": "2",
+ "expression": "Find senders containing apache[any char]org",
+ "explanation": "apache.org",
+ "targetsSender": true,
+ "targetsRecipients": false,
+ "targetsContent": false
+ }
+]}
+....
+
+=== Store DLP configuration by sender domain
+
+Store a DLP configuration for corresponding `senderDomain`, if any item
+of DLP configuration in the request is stored before, it will not be
+stored anymore
+
+....
+curl -XPUT http://ip:port/dlp/rules/{senderDomain}
+....
+
+The body can contain a list of DLP configuration items formed by those
+fields: - `id`(String) is mandatory, unique identifier of the
+configuration item - `expression`(String) is mandatory, regular
+expression to match contents of targets - `explanation`(String) is
+optional, description of the configuration item -
+`targetsSender`(boolean) is optional and defaults to false. If true,
+`expression` will be applied to Sender and to From headers of the mail -
+`targetsContent`(boolean) is optional and defaults to false. If true,
+`expression` will be applied to Subject headers and textual bodies
+(text/plain and text/html) of the mail - `targetsRecipients`(boolean) is
+optional and defaults to false. If true, `expression` will be applied to
+recipients of the mail
+
+This is an example of returned body. The rules field is a list of rules
+as described below.
+
+....
+{"rules": [
+ {
+ "id": "1",
+ "expression": "james.org",
+ "explanation": "Find senders or recipients containing james[any char]org",
+ "targetsSender": true,
+ "targetsRecipients": true,
+ "targetsContent": false
+ },
+ {
+ "id": "2",
+ "expression": "Find senders containing apache[any char]org",
+ "explanation": "apache.org",
+ "targetsSender": true,
+ "targetsRecipients": false,
+ "targetsContent": false
+ }
+]}
+....
+
+Response codes:
+
+* 204: List of dlp configuration items is stored
+* 400: Invalid `senderDomain` or payload in request
+* 404: The domain does not exist.
+
+=== Remove DLP configuration by sender domain
+
+Remove a DLP configuration for corresponding `senderDomain`
+
+....
+curl -XDELETE http://ip:port/dlp/rules/{senderDomain}
+....
+
+Response codes:
+
+* 204: DLP configuration is removed
+* 400: Invalid `senderDomain` or payload in request
+* 404: The domain does not exist.
+
+=== Fetch a DLP configuration item by sender domain and rule id
+
+Retrieve a DLP configuration rule for corresponding `senderDomain` and a
+`ruleId`
+
+....
+curl -XGET http://ip:port/dlp/rules/{senderDomain}/rules/{ruleId}
+....
+
+Response codes:
+
+* 200: A dlp configuration item is returned
+* 400: Invalid `senderDomain` or payload in request
+* 404: The domain and/or the rule does not exist.
+
+This is an example of returned body.
+
+....
+{
+ "id": "1",
+ "expression": "james.org",
+ "explanation": "Find senders or recipients containing james[any char]org",
+ "targetsSender": true,
+ "targetsRecipients": true,
+ "targetsContent": false
+}
+....
+
+== Administrating Sieve quotas
+
+Some limitations on space Users Sieve script can occupy can be
+configured by default, and overridden by user.
+
+=== Retrieving global sieve quota
+
+This endpoints allows to retrieve the global Sieve quota, which will be
+users default:
+
+....
+curl -XGET http://ip:port/sieve/quota/default
+....
+
+Will return the bytes count allowed by user per default on this server.
+
+....
+102400
+....
+
+Response codes:
+
+* 200: Request is a success and the value is returned
+* 204: No default quota is being configured
+
+=== Updating global sieve quota
+
+This endpoints allows to update the global Sieve quota, which will be
+users default:
+
+....
+curl -XPUT http://ip:port/sieve/quota/default
+....
+
+With the body being the bytes count allowed by user per default on this
+server.
+
+....
+102400
+....
+
+Response codes:
+
+* 204: Operation succeeded
+* 400: Invalid payload
+
+=== Removing global sieve quota
+
+This endpoints allows to remove the global Sieve quota. There will no
+more be users default:
+
+....
+curl -XDELETE http://ip:port/sieve/quota/default
+....
+
+Response codes:
+
+* 204: Operation succeeded
+
+=== Retrieving user sieve quota
+
+This endpoints allows to retrieve the Sieve quota of a user, which will
+be this users quota:
+
+....
+curl -XGET http://ip:port/sieve/quota/users/user@domain.com
+....
+
+Will return the bytes count allowed for this user.
+
+....
+102400
+....
+
+Response codes:
+
+* 200: Request is a success and the value is returned
+* 204: No quota is being configured for this user
+
+=== Updating user sieve quota
+
+This endpoints allows to update the Sieve quota of a user, which will be
+users default:
+
+....
+curl -XPUT http://ip:port/sieve/quota/users/user@domain.com
+....
+
+With the body being the bytes count allowed for this user on this
+server.
+
+....
+102400
+....
+
+Response codes:
+
+* 204: Operation succeeded
+* 400: Invalid payload
+
+=== Removing user sieve quota
+
+This endpoints allows to remove the Sieve quota of a user. There will no
+more quota for this user:
+
+....
+curl -XDELETE http://ip:port/sieve/quota/users/user@domain.com
+....
+
+Response codes:
+
+* 204: Operation succeeded
+
+== Event Dead Letter
+
+The EventBus allows to register `group listeners' that are called in a
+distributed fashion. These group listeners enable the implementation of
+some advanced mailbox manager feature like indexing, spam reporting,
+quota management and the like.
+
+Upon exceptions, a bounded number of retries are performed (with
+exponential backoff delays). If after those retries the listener is
+still failing, then the event will be stored in the ``Event Dead
+Letter''. This API allows diagnosing issues, as well as performing event
+replay.
+
+=== Listing mailbox listener groups
+
+This endpoint allows discovering the list of mailbox listener groups.
+
+....
+curl -XGET http://ip:port/events/deadLetter/groups
+....
+
+Will return a list of group names that can be further used to interact
+with the dead letter API:
+
+....
+["org.apache.james.mailbox.events.EventBusTestFixture$GroupA", "org.apache.james.mailbox.events.GenericGroup-abc"]
+....
+
+Response codes:
+
+* 200: Success. A list of group names is returned.
+
+=== Listing failed events
+
+This endpoint allows listing failed events for a given group:
+
+....
+curl -XGET http://ip:port/events/deadLetter/groups/org.apache.james.mailbox.events.EventBusTestFixture$GroupA
+....
+
+Will return a list of insertionIds:
+
+....
+["6e0dd59d-660e-4d9b-b22f-0354479f47b4", "58a8f59d-660e-4d9b-b22f-0354486322a2"]
+....
+
+Response codes:
+
+* 200: Success. A list of insertion ids is returned.
+* 400: Invalid group name
+
+=== Getting event details
+
+....
+curl -XGET http://ip:port/events/deadLetter/groups/org.apache.james.mailbox.events.EventBusTestFixture$GroupA/6e0dd59d-660e-4d9b-b22f-0354479f47b4
+....
+
+Will return the full JSON associated with this event.
+
+Response codes:
+
+* 200: Success. A JSON representing this event is returned.
+* 400: Invalid group name or `insertionId`
+* 404: No event with this `insertionId`
+
+=== Deleting an event
+
+....
+curl -XDELETE http://ip:port/events/deadLetter/groups/org.apache.james.mailbox.events.EventBusTestFixture$GroupA/6e0dd59d-660e-4d9b-b22f-0354479f47b4
+....
+
+Will delete this event.
+
+Response codes:
+
+* 204: Success
+* 400: Invalid group name or `insertionId`
+
+=== Redeliver all events
+
+....
+curl -XPOST http://ip:port/events/deadLetter?action=redeliver
+....
+
+Will create a task that will attempt to redeliver all events stored in
+``Event Dead Letter''. If successful, redelivered events will then be
+removed from ``Dead Letter''.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: the taskId of the created task
+* 400: Invalid action argument
+
+=== Redeliver group events
+
+....
+curl -XPOST http://ip:port/events/deadLetter/groups/org.apache.james.mailbox.events.EventBusTestFixture$GroupA
+....
+
+Will create a task that will attempt to redeliver all events of a
+particular group stored in ``Event Dead Letter''. If successful,
+redelivered events will then be removed from ``Dead Letter''.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: the taskId of the created task
+* 400: Invalid group name or action argument
+
+=== Redeliver a single event
+
+....
+curl -XPOST http://ip:port/events/deadLetter/groups/org.apache.james.mailbox.events.EventBusTestFixture$GroupA/6e0dd59d-660e-4d9b-b22f-0354479f47b4?action=reDeliver
+....
+
+Will create a task that will attempt to redeliver a single event of a
+particular group stored in ``Event Dead Letter''. If successful,
+redelivered event will then be removed from ``Dead Letter''.
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes:
+
+* 201: the taskId of the created task
+* 400: Invalid group name, insertion id or action argument
+* 404: No event with this insertionId
+
+== Deleted Messages Vault
+
+The `Deleted Message Vault plugin' allows you to keep users deleted
+messages during a given retention time. This set of routes allow you to
+_restore_ users deleted messages or export them in an archive.
+
+To move deleted messages in the vault, you need to specifically
+configure the DeletedMessageVault PreDeletionHook.
+
+=== Restore Deleted Messages
+
+Deleted messages of a specific user can be restored by calling the
+following endpoint:
+
+....
+curl -XPOST http://ip:port/deletedMessages/users/userToRestore@domain.ext?action=restore
+
+{
+ "combinator": "and",
+ "criteria": [
+ {
+ "fieldName": "subject",
+ "operator": "containsIgnoreCase",
+ "value": "Apache James"
+ },
+ {
+ "fieldName": "deliveryDate",
+ "operator": "beforeOrEquals",
+ "value": "2014-10-30T14:12:00Z"
+ },
+ {
+ "fieldName": "deletionDate",
+ "operator": "afterOrEquals",
+ "value": "2015-10-20T09:08:00Z"
+ },
+ {
+ "fieldName": "recipients","
+ "operator": "contains","
+ "value": "recipient@james.org"
+ },
+ {
+ "fieldName": "hasAttachment",
+ "operator": "equals",
+ "value": "false"
+ },
+ {
+ "fieldName": "sender",
+ "operator": "equals",
+ "value": "sender@apache.org"
+ },
+ {
+ "fieldName": "originMailboxes",
+ "operator": "contains",
+ "value": "02874f7c-d10e-102f-acda-0015176f7922"
+ }
+ ]
+};
+....
+
+The requested Json body is made from a list of criterion objects which
+have the following structure:
+
+....
+{
+ "fieldName": "supportedFieldName",
+ "operator": "supportedOperator",
+ "value": "A plain string representing the matching value of the corresponding field"
+}
+....
+
+Deleted Messages which are matched with the *all* criterion in the query
+body will be restored. Here are a list of supported fieldName for the
+restoring:
+
+* subject: represents for deleted message `subject` field matching.
+Supports below string operators:
+** contains
+** containsIgnoreCase
+** equals
+** equalsIgnoreCase
+* deliveryDate: represents for deleted message `deliveryDate` field
+matching. Tested value should follow the right date time with zone
+offset format (ISO-8601) like `2008-09-15T15:53:00+05:00` or
+`2008-09-15T15:53:00Z` Supports below date time operators:
+** beforeOrEquals: is the deleted message’s `deliveryDate` before or
+equals the time of tested value.
+** afterOrEquals: is the deleted message’s `deliveryDate` after or
+equals the time of tested value
+* deletionDate: represents for deleted message `deletionDate` field
+matching. Tested value & Supports operators: similar to `deliveryDate`
+* sender: represents for deleted message `sender` field matching. Tested
+value should be a valid mail address. Supports mail address operator:
+** equals: does the tested sender equal to the sender of the tested
+deleted message ? +
+* recipients: represents for deleted message `recipients` field
+matching. Tested value should be a valid mail address. Supports list
+mail address operator:
+** contains: does the tested deleted message’s recipients contain tested
+recipient ?
+* hasAttachment: represents for deleted message `hasAttachment` field
+matching. Tested value could be `false` or `true`. Supports boolean
+operator:
+** equals: does the tested deleted message’s hasAttachment property
+equal to the tested hasAttachment value?
+* originMailboxes: represents for deleted message `originMailboxes`
+field matching. Tested value is a string serialized of mailbox id.
+Supports list mailbox id operators:
+** contains: does the tested deleted message’s originMailbox ids contain
+tested mailbox id ?
+
+Messages in the Deleted Messages Vault of a specified user that are
+matched with Query Json Object in the body will be appended to his
+`Restored-Messages' mailbox, which will be created if needed.
+
+*Note*:
+
+* Query parameter `action` is required and should have the value
+`restore` to represent the restoring feature. Otherwise, a bad request
+response will be returned
+* Query parameter `action` is case sensitive
+* fieldName & operator passed to the routes are case sensitive
+* Currently, we only support query combinator `and` value, otherwise,
+requests will be rejected
+* If you only want to restore by only one criterion, the json body could
+be simplified to a single criterion:
+
+....
+{
+ "fieldName": "subject",
+ "operator": "containsIgnoreCase",
+ "value": "Apache James"
+}
+....
+
+* For restoring all deleted messages, passing a query json with an empty
+criterion list to represent `matching all deleted messages`:
+
+....
+{
+ "combinator": "and",
+ "criteria": []
+}
+....
+
+*Warning*: Current web-admin uses `US` locale as the default. Therefore,
+there might be some conflicts when using String `containsIgnoreCase`
+comparators to apply on the String data of other special locales stored
+in the Vault. More details at
+https://issues.apache.org/jira/browse/MAILBOX-384[JIRA]
+
+Response code:
+
+* 201: Task for restoring deleted has been created
+* 400: Bad request:
+** action query param is not present
+** action query param is not a valid action
+** user parameter is invalid
+** can not parse the JSON body
+** Json query object contains unsupported operator, fieldName
+** Json query object values violate parsing rules
+* 404: User not found
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+The scheduled task will have the following type
+`deleted-messages-restore` and the following `additionalInformation`:
+
+....
+{
+ "successfulRestoreCount": 47,
+ "errorRestoreCount": 0,
+ "user": "userToRestore@domain.ext"
+}
+....
+
+while:
+
+* successfulRestoreCount: number of restored messages
+* errorRestoreCount: number of messages that failed to restore
+* user: owner of deleted messages need to restore
+
+=== Export Deleted Messages
+
+Retrieve deleted messages matched with requested query from an user then
+share the content to a targeted mail address (exportTo)
+
+....
+curl -XPOST 'http://ip:port/deletedMessages/users/userExportFrom@domain.ext?action=export&exportTo=userReceiving@domain.ext'
+
+BODY: is the json query has the same structure with Restore Deleted Messages section
+....
+
+*Note*: Json query passing into the body follows the same rules &
+restrictions like in link:#Restore_deleted_messages[Restore Deleted
+Messages]
+
+Response code:
+
+* 201: Task for exporting has been created
+* 400: Bad request:
+** exportTo query param is not present
+** exportTo query param is not a valid mail address
+** action query param is not present
+** action query param is not a valid action
+** user parameter is invalid
+** can not parse the JSON body
+** Json query object contains unsupported operator, fieldName
+** Json query object values violate parsing rules
+* 404: User not found
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+The scheduled task will have the following type
+`deleted-messages-export` and the following `additionalInformation`:
+
+....
+{
+ "userExportFrom": "userToRestore@domain.ext",
+ "exportTo": "userReceiving@domain.ext",
+ "totalExportedMessages": 1432
+}
+....
+
+while:
+
+ * userExportFrom: export deleted messages from this user
+ * exportTo: content of deleted messages have been shared to this mail
+address
+ * totalExportedMessages: number of deleted messages match with
+json query, then being shared to sharee.
+
+=== Purge Deleted Messages
+
+You can overwrite `retentionPeriod' configuration in
+`deletedMessageVault' configuration file or use the default value of 1
+year.
+
+Purge all deleted messages older than the configured `retentionPeriod'
+
+....
+curl -XDELETE http://ip:port/deletedMessages?scope=expired
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response code:
+
+* 201: Task for purging has been created
+* 400: Bad request:
+** action query param is not present
+** action query param is not a valid action
+
+You may want to call this endpoint on a regular basis.
+
+=== Permanently Remove Deleted Message
+
+Delete a Deleted Message with `MessageId`
+
+....
+curl -XDELETE http://ip:port/deletedMessages/users/user@domain.ext/messages/3294a976-ce63-491e-bd52-1b6f465ed7a2
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response code:
+
+* 201: Task for deleting message has been created
+* 400: Bad request:
+** user parameter is invalid
+** messageId parameter is invalid
+* 404: User not found
+
+The scheduled task will have the following type
+`deleted-messages-delete` and the following `additionalInformation`:
+
+....
+ {
+ "userName": "user@domain.ext",
+ "messageId": "3294a976-ce63-491e-bd52-1b6f465ed7a2"
+ }
+....
+
+while: - user: delete deleted messages from this user - deleteMessageId:
+messageId of deleted messages will be delete
+
+== Task management
+
+Some webadmin features schedules tasks. The task management API allow to
+monitor and manage the execution of the following tasks.
+
+Note that the `taskId` used in the following APIs is returned by other
+WebAdmin APIs scheduling tasks.
+
+=== Getting a task details
+
+....
+curl -XGET http://ip:port/tasks/3294a976-ce63-491e-bd52-1b6f465ed7a2
+....
+
+An Execution Report will be returned:
+
+....
+{
+ "submitDate": "2017-12-27T15:15:24.805+0700",
+ "startedDate": "2017-12-27T15:15:24.809+0700",
+ "completedDate": "2017-12-27T15:15:24.815+0700",
+ "cancelledDate": null,
+ "failedDate": null,
+ "taskId": "3294a976-ce63-491e-bd52-1b6f465ed7a2",
+ "additionalInformation": {},
+ "status": "completed",
+ "type": "type-of-the-task"
+}
+....
+
+Note that:
+
+* `status` can have the value:
+** `waiting`: The task is scheduled but its execution did not start yet
+** `inProgress`: The task is currently executed
+** `cancelled`: The task had been cancelled
+** `completed`: The task execution is finished, and this execution is a
+success
+** `failed`: The task execution is finished, and this execution is a
+failure
+* `additionalInformation` is a task specific object giving additional
+information and context about that task. The structure of this
+`additionalInformation` field is provided along the specific task
+submission endpoint.
+
+Response codes:
+
+* 200: The specific task was found and the execution report exposed
+above is returned
+* 400: Invalid task ID
+* 404: Task ID was not found
+
+=== Awaiting a task
+
+One can await the end of a task, then receive it’s final execution
+report.
+
+That feature is especially usefull for testing purpose but still can
+serve real-life scenari.
+
+....
+curl -XGET http://ip:port/tasks/3294a976-ce63-491e-bd52-1b6f465ed7a2/await?timeout=duration
+....
+
+An Execution Report will be returned.
+
+`timeout` is optional. By default it is set to 365 days (the maximum
+value). The expected value is expressed in the following format:
+`Nunit`. `N` should be strictly positive. `unit` could be either in the
+short form (`s`, `m`, `h`, etc.), or in the long form (`day`, `week`,
+`month`, etc.).
+
+Examples:
+
+* `30s`
+* `5m`
+* `7d`
+* `1y`
+
+Response codes:
+
+* 200: The specific task was found and the execution report exposed
+above is returned
+* 400: Invalid task ID or invalid timeout
+* 404: Task ID was not found
+* 408: The timeout has been reached
+
+=== Cancelling a task
+
+You can cancel a task by calling:
+
+....
+curl -XDELETE http://ip:port/tasks/3294a976-ce63-491e-bd52-1b6f465ed7a2
+....
+
+Response codes:
+
+* 204: Task had been cancelled
+* 400: Invalid task ID
+
+=== Listing tasks
+
+A list of all tasks can be retrieved:
+
+....
+curl -XGET http://ip:port/tasks
+....
+
+Will return a list of Execution reports
+
+One can filter the above results by status. For example:
+
+....
+curl -XGET http://ip:port/tasks?status=inProgress
+....
+
+Will return a list of Execution reports that are currently in progress.
+
+Response codes:
+
+* 200: A list of corresponding tasks is returned
+* 400: Invalid status value
+
+=== Endpoints returning a task
+
+Many endpoints do generate a task.
+
+Example:
+
+....
+curl -XPOST /endpoint?action={action}
+....
+
+The response to these requests will be the scheduled `taskId` :
+
+....
+{"taskId":"5641376-02ed-47bd-bcc7-76ff6262d92a"}
+....
+
+Positionned headers:
+
+* Location header indicates the location of the resource associated with
+the scheduled task. Example:
+
+....
+Location: /tasks/3294a976-ce63-491e-bd52-1b6f465ed7a2
+....
+
+Response codes:
+
+* 201: Task generation succeeded. Corresponding task id is returned.
+* Other response codes might be returned depending on the endpoint
+
+The additional information returned depends on the scheduled task type
+and is documented in the endpoint documentation.
+
+== Cassandra extra operations
+
+Some webadmin features to manage some extra operations on Cassandra
+tables, like solving inconsistencies on projection tables. Such
+inconsistencies can be for example created by a fail of the DAO to add a
+mapping into
+’mappings_sources`, while it was successful regarding the`rrt` table.
+
+=== Operations on mappings sources
+
+You can do a series of action on `mappings_sources` projection table :
+
+....
+curl -XPOST /cassandra/mappings?action={action}
+....
+
+Will return the taskId corresponding to the related task. Actions
+supported so far are :
+
+* SolveInconsistencies : cleans up first all the mappings in
+`mappings_sources` index and then repopulate it correctly. In the
+meantime, listing sources of a mapping might create temporary
+inconsistencies during the process.
+
+For example :
+
+....
+curl -XPOST /cassandra/mappings?action=SolveInconsistencies
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning
+a task].
+
+Response codes :
+
+* 201: the taskId of the created task
+* 400: Invalid action argument for performing operation on mappings data
\ No newline at end of file
diff --git a/docs/modules/servers/pages/distributed/run-docker.adoc b/docs/modules/servers/pages/distributed/run-docker.adoc
index d3fd897..2646ae1 100644
--- a/docs/modules/servers/pages/distributed/run-docker.adoc
+++ b/docs/modules/servers/pages/distributed/run-docker.adoc
@@ -1,6 +1,36 @@
= Run with docker
-== Requirements
+== Running via docker-compose
+
+
+Requirements: docker & docker-compose installed.
+
+When you try James this way, you will use the most current state of James.
+It will be configured to run with Cassandra & ElasticSearch.
+All those three components will be started with a single command.
+
+You can retrieve the docker-compose file :
+
+ $ wget https://raw.githubusercontent.com/apache/james-project/master/dockerfiles/run/docker-compose.yml
+
+Then, you just have to start the services:
+
+ $ docker-compose up
+
+Wait a few seconds in order to have all those services start up. You will see the following log when James is available:
+james | Started : true
+
+A default domain, james.local, has been created. You can see this by running:
+
+ $ docker exec james java -jar /root/james-cli.jar -h 127.0.0.1 -p 9999 listdomains
+
+James will respond to IMAP port 143 and SMTP port 25.
+You have to create users before playing with james. You may also want to create other domains.
+Follow the 'Useful commands' section for more information about James CLI.
+
+== Run with docker
+
+=== Requirements
Built artifacts should be in ./dockerfiles/run/guice/cassandra-rabbitmq/destination folder for cassandra.
If you haven't already:
@@ -10,7 +40,7 @@ If you haven't already:
-v $PWD/dockerfiles/run/guice/cassandra-rabbitmq/destination:/cassandra-rabbitmq/destination \
-t james/project -s HEAD
-== Running
+=== Running
You need a running *cassandra* in docker. To achieve this run:
@@ -62,7 +92,7 @@ If you want to pass additional options to the underlying java command, you can c
To have log file accessible on a volume, add *-v $PWD/logs:/logs* option to the above command line, where *$PWD/logs* is your local directory to put files in.
-== Instrumentation
+=== Instrumentation
You can use Glowroot to instrumentalize James. The provided guice docker files allow a simple way to do it.
In order to activate Glowroot you need to run the container with the environment variable _GLOWROOT_ACTIVATED_ set to _true_
and to expose the glowroot instrumentation ui port.
@@ -76,7 +106,7 @@ See the https://github.com/glowroot/glowroot/wiki/Agent-Installation-(with-Embed
Or by mapping the 4000 port to the IP of the desired network interface, for example `-p 127.0.0.1:4000:4000`.
-== Handling attachment indexing
+=== Handling attachment indexing
You can handle attachment text extraction before indexing in ElasticSearch. This makes attachments searchable. To enable this:
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org