You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by bt...@apache.org on 2020/06/25 02:56:06 UTC
[james-project] branch master updated (56618ac -> 19815ae)
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from 56618ac JAMES-3029 add test to check that mail is well received with 10 recipients
new bd9147a JAMES-3184 Transform throttle into a reusable transformation
new d666723 JAMES-3202 `ReindexingRunningOptionsParser.parse` is a static method, can call it directly
new 1727a0e JAMES-3202 Add reindexing mode in RunningOptions
new 992cec4 JAMES-3202 Reindex only outdated documents with the Mode option set to CORRECT in reindexing tasks
new bdfdeb6 JAMES-3202 Integration tests for Reindex with Mode set to Correct
new 54722ea JAMES-3202 Update webadmin documentation with the new Mode parameter in RunningOptions for reindexation tasks
new 0ba8f1e JAMES-3255 Postmaster example values should be valid
new 6d2cf13 JAMES-3213 Change ICALToJsonAttribute replyTo header source to Reply-To
new 4003c37 JAMES-3265 Impement a MessageMapper method to reset all recents
new 19815ae JAMES-3268 do not desactivate MOVE and ACL capabilities in docker images
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../cassandra-ldap/destination/conf/imapserver.xml | 2 -
.../destination/conf/mailetcontainer.xml | 4 +-
.../destination/conf/imapserver.xml | 2 -
.../destination/conf/mailetcontainer.xml | 4 +-
.../destination/conf/imapserver.xml | 2 -
.../destination/conf/mailetcontainer.xml | 4 +-
.../cassandra/destination/conf/imapserver.xml | 2 -
.../cassandra/destination/conf/mailetcontainer.xml | 4 +-
.../jpa-smtp/destination/conf/mailetcontainer.xml | 4 +-
.../run/guice/jpa/destination/conf/imapserver.xml | 2 -
.../guice/jpa/destination/conf/mailetcontainer.xml | 4 +-
.../guice/memory/destination/conf/imapserver.xml | 2 -
.../memory/destination/conf/mailetcontainer.xml | 4 +-
.../run/spring/destination/conf/imapserver.xml | 2 -
.../spring/destination/conf/mailetcontainer.xml | 4 +-
.../src/main/resources/mailetcontainer.xml | 2 +-
.../apache/james/mailbox/indexer/ReIndexer.java | 81 +++-
.../cassandra/mail/CassandraMessageMapper.java | 21 +-
.../task/SolveMessageInconsistenciesService.java | 24 +-
.../james/mailbox/store/StoreMessageManager.java | 43 ++-
.../james/mailbox/store/mail/MessageMapper.java | 19 +
.../store/mail/model/MessageMapperTest.java | 33 +-
.../mailbox/tools/indexer/ReIndexerPerformer.java | 112 ++++--
.../mailbox/tools/indexer/RunningOptionsDTO.java | 18 +-
...rorRecoveryIndexationTaskSerializationTest.java | 17 +-
.../FullReindexingTaskSerializationTest.java | 9 +-
...ngleMailboxReindexingTaskSerializationTest.java | 9 +-
.../UserReindexingTaskSerializationTest.java | 9 +-
.../quota/task/RecomputeCurrentQuotasService.java | 10 +-
.../transport/mailets/ICALToJsonAttribute.java | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
server/app/src/main/resources/mailetcontainer.xml | 4 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../sample-configuration/mailetcontainer.xml | 4 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../sample-configuration/mailetcontainer.xml | 4 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../java/org/apache/james/util/ReactorUtils.java | 55 ++-
.../org/apache/james/util/ReactorUtilsTest.java | 62 ++-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../jmap/MessageFastViewProjectionCorrector.java | 7 +-
server/protocols/webadmin/webadmin-mailbox/pom.xml | 18 +-
.../james/webadmin/routes/MailboxesRoutes.java | 2 +-
.../routes/ReindexingRunningOptionsParser.java | 11 +-
.../james/webadmin/routes/MailboxesRoutesTest.java | 423 ++++++++++++++++++++-
.../webadmin/routes/UserMailboxesRoutesTest.java | 259 ++++++++++++-
src/homepage/howTo/mail-processing.html | 2 +-
src/site/markdown/server/manage-webadmin.md | 85 ++++-
61 files changed, 1194 insertions(+), 233 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 10/10: JAMES-3268 do not desactivate MOVE and ACL
capabilities in docker images
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 19815ae6bb7891a7745f193847afd75513f4aada
Author: Rémi KOWALSKI <rk...@linagora.com>
AuthorDate: Tue Jun 23 16:28:01 2020 +0200
JAMES-3268 do not desactivate MOVE and ACL capabilities in docker images
---
dockerfiles/run/guice/cassandra-ldap/destination/conf/imapserver.xml | 2 --
.../run/guice/cassandra-rabbitmq-ldap/destination/conf/imapserver.xml | 2 --
.../run/guice/cassandra-rabbitmq/destination/conf/imapserver.xml | 2 --
dockerfiles/run/guice/cassandra/destination/conf/imapserver.xml | 2 --
dockerfiles/run/guice/jpa/destination/conf/imapserver.xml | 2 --
dockerfiles/run/guice/memory/destination/conf/imapserver.xml | 2 --
dockerfiles/run/spring/destination/conf/imapserver.xml | 2 --
7 files changed, 14 deletions(-)
diff --git a/dockerfiles/run/guice/cassandra-ldap/destination/conf/imapserver.xml b/dockerfiles/run/guice/cassandra-ldap/destination/conf/imapserver.xml
index 48e552c..12aa52d 100644
--- a/dockerfiles/run/guice/cassandra-ldap/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/cassandra-ldap/destination/conf/imapserver.xml
@@ -38,7 +38,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -57,7 +56,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/imapserver.xml b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/imapserver.xml
index 48e552c..12aa52d 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/imapserver.xml
@@ -38,7 +38,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -57,7 +56,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/imapserver.xml b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/imapserver.xml
index 48e552c..12aa52d 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/imapserver.xml
@@ -38,7 +38,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -57,7 +56,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/guice/cassandra/destination/conf/imapserver.xml b/dockerfiles/run/guice/cassandra/destination/conf/imapserver.xml
index 48e552c..12aa52d 100644
--- a/dockerfiles/run/guice/cassandra/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/cassandra/destination/conf/imapserver.xml
@@ -38,7 +38,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -57,7 +56,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/guice/jpa/destination/conf/imapserver.xml b/dockerfiles/run/guice/jpa/destination/conf/imapserver.xml
index a1306fa..72f1906 100644
--- a/dockerfiles/run/guice/jpa/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/jpa/destination/conf/imapserver.xml
@@ -39,7 +39,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -58,7 +57,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/guice/memory/destination/conf/imapserver.xml b/dockerfiles/run/guice/memory/destination/conf/imapserver.xml
index a1306fa..72f1906 100644
--- a/dockerfiles/run/guice/memory/destination/conf/imapserver.xml
+++ b/dockerfiles/run/guice/memory/destination/conf/imapserver.xml
@@ -39,7 +39,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -58,7 +57,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
diff --git a/dockerfiles/run/spring/destination/conf/imapserver.xml b/dockerfiles/run/spring/destination/conf/imapserver.xml
index 6616d24..310625d 100644
--- a/dockerfiles/run/spring/destination/conf/imapserver.xml
+++ b/dockerfiles/run/spring/destination/conf/imapserver.xml
@@ -37,7 +37,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
<imapserver enabled="true">
@@ -56,7 +55,6 @@ under the License.
<connectionLimitPerIP>0</connectionLimitPerIP>
<idleTimeInterval>120</idleTimeInterval>
<idleTimeIntervalUnit>SECONDS</idleTimeIntervalUnit>
- <disabledCaps>ACL|MOVE</disabledCaps>
<enableIdle>true</enableIdle>
</imapserver>
</imapservers>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 09/10: JAMES-3265 Impement a MessageMapper method
to reset all recents
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4003c37e9ffb87902124f8109ed4c1acd12e575c
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue Jun 23 14:11:21 2020 +0700
JAMES-3265 Impement a MessageMapper method to reset all recents
This enables a single modseq allocation for the entire operaton,
fasting significantly things up.
Cassandra query analysis showed that a large portion of time was
spent upon some IMAP SELECT operations updating modseqs.
---
.../cassandra/mail/CassandraMessageMapper.java | 21 ++++++++++-
.../james/mailbox/store/StoreMessageManager.java | 43 +++++++++++++---------
.../james/mailbox/store/mail/MessageMapper.java | 19 ++++++++++
.../store/mail/model/MessageMapperTest.java | 33 ++++++++++++++++-
4 files changed, 97 insertions(+), 19 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java
index 39a7757..82c9e6b 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapper.java
@@ -35,6 +35,7 @@ import org.apache.commons.lang3.tuple.Pair;
import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration;
import org.apache.james.mailbox.ApplicableFlagBuilder;
import org.apache.james.mailbox.FlagsBuilder;
+import org.apache.james.mailbox.MessageManager.FlagsUpdateMode;
import org.apache.james.mailbox.MessageUid;
import org.apache.james.mailbox.ModSeq;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
@@ -294,12 +295,30 @@ public class CassandraMessageMapper implements MessageMapper {
Flux<ComposedMessageIdWithMetaData> toBeUpdated = messageIdDAO.retrieveMessages(mailboxId, range, Limit.unlimited());
+ return updateFlags(flagUpdateCalculator, mailboxId, toBeUpdated).iterator();
+ }
+
+ private List<UpdatedFlags> updateFlags(FlagsUpdateCalculator flagUpdateCalculator, CassandraId mailboxId, Flux<ComposedMessageIdWithMetaData> toBeUpdated) {
FlagsUpdateStageResult firstResult = runUpdateStage(mailboxId, toBeUpdated, flagUpdateCalculator).block();
FlagsUpdateStageResult finalResult = handleUpdatesStagedRetry(mailboxId, flagUpdateCalculator, firstResult);
if (finalResult.containsFailedResults()) {
LOGGER.error("Can not update following UIDs {} for mailbox {}", finalResult.getFailed(), mailboxId.asUuid());
}
- return finalResult.getSucceeded().iterator();
+ return finalResult.getSucceeded();
+ }
+
+ @Override
+ public List<UpdatedFlags> resetRecent(Mailbox mailbox) {
+ CassandraId mailboxId = (CassandraId) mailbox.getMailboxId();
+
+ Flux<ComposedMessageIdWithMetaData> toBeUpdated = mailboxRecentDAO.getRecentMessageUidsInMailbox(mailboxId)
+ .collectList()
+ .flatMapMany(uids -> Flux.fromIterable(MessageRange.toRanges(uids)))
+ .concatMap(range -> messageIdDAO.retrieveMessages(mailboxId, range, Limit.unlimited()))
+ .filter(message -> message.getFlags().contains(Flag.RECENT));
+ FlagsUpdateCalculator calculator = new FlagsUpdateCalculator(new Flags(Flag.RECENT), FlagsUpdateMode.REMOVE);
+
+ return updateFlags(calculator, mailboxId, toBeUpdated);
}
private FlagsUpdateStageResult handleUpdatesStagedRetry(CassandraId mailboxId, FlagsUpdateCalculator flagUpdateCalculator, FlagsUpdateStageResult firstResult) {
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
index d2f6520..a90e81b 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageManager.java
@@ -656,30 +656,39 @@ public class StoreMessageManager implements MessageManager {
* the recent flag on the messages for the uids
*/
protected List<MessageUid> recent(final boolean reset, MailboxSession mailboxSession) throws MailboxException {
- if (reset) {
- if (!isWriteable(mailboxSession)) {
- throw new ReadOnlyException(getMailboxPath());
- }
- }
- final MessageMapper messageMapper = mapperFactory.getMessageMapper(mailboxSession);
+ MessageMapper messageMapper = mapperFactory.getMessageMapper(mailboxSession);
return messageMapper.execute(() -> {
- final List<MessageUid> members = messageMapper.findRecentMessageUidsInMailbox(getMailboxEntity());
-
- // Convert to MessageRanges so we may be able to optimize the
- // flag update
- List<MessageRange> ranges = MessageRange.toRanges(members);
- for (MessageRange range : ranges) {
- if (reset) {
- // only call save if we need to
- messageMapper.updateFlags(getMailboxEntity(), new FlagsUpdateCalculator(new Flags(Flag.RECENT), FlagsUpdateMode.REMOVE), range);
- }
+ if (reset) {
+ return resetRecents(messageMapper, mailboxSession);
}
- return members;
+ return messageMapper.findRecentMessageUidsInMailbox(getMailboxEntity());
});
}
+ private List<MessageUid> resetRecents(MessageMapper messageMapper, MailboxSession mailboxSession) throws MailboxException {
+ if (!isWriteable(mailboxSession)) {
+ throw new ReadOnlyException(getMailboxPath());
+ }
+
+ List<UpdatedFlags> updatedFlags = messageMapper.resetRecent(getMailboxEntity());
+
+ eventBus.dispatch(EventFactory.flagsUpdated()
+ .randomEventId()
+ .mailboxSession(mailboxSession)
+ .mailbox(getMailboxEntity())
+ .updatedFlags(updatedFlags)
+ .build(),
+ new MailboxIdRegistrationKey(mailbox.getMailboxId()))
+ .subscribeOn(Schedulers.elastic())
+ .block();
+
+ return updatedFlags.stream()
+ .map(UpdatedFlags::getUid)
+ .collect(Guavate.toImmutableList());
+ }
+
private void runPredeletionHooks(List<MessageUid> uids, MailboxSession session) {
MessageMapper messageMapper = mapperFactory.getMessageMapper(session);
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageMapper.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageMapper.java
index 2c79390..ece0c7d 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageMapper.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageMapper.java
@@ -18,6 +18,8 @@
****************************************************************/
package org.apache.james.mailbox.store.mail;
+import static javax.mail.Flags.Flag.RECENT;
+
import java.util.Iterator;
import java.util.List;
import java.util.Map;
@@ -25,6 +27,7 @@ import java.util.Optional;
import javax.mail.Flags;
+import org.apache.james.mailbox.MessageManager.FlagsUpdateMode;
import org.apache.james.mailbox.MessageUid;
import org.apache.james.mailbox.ModSeq;
import org.apache.james.mailbox.exception.MailboxException;
@@ -39,6 +42,8 @@ import org.apache.james.mailbox.store.mail.model.Property;
import org.apache.james.mailbox.store.transaction.Mapper;
import org.apache.james.util.streams.Iterators;
+import com.google.common.collect.ImmutableList;
+
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
@@ -121,6 +126,20 @@ public interface MessageMapper extends Mapper {
*/
Iterator<UpdatedFlags> updateFlags(Mailbox mailbox, FlagsUpdateCalculator flagsUpdateCalculator,
final MessageRange set) throws MailboxException;
+
+ default List<UpdatedFlags> resetRecent(Mailbox mailbox) throws MailboxException {
+ final List<MessageUid> members = findRecentMessageUidsInMailbox(mailbox);
+ ImmutableList.Builder<UpdatedFlags> result = ImmutableList.builder();
+
+ FlagsUpdateCalculator calculator = new FlagsUpdateCalculator(new Flags(RECENT), FlagsUpdateMode.REMOVE);
+ // Convert to MessageRanges so we may be able to optimize the flag update
+ List<MessageRange> ranges = MessageRange.toRanges(members);
+ for (MessageRange range : ranges) {
+ result.addAll(updateFlags(mailbox, calculator, range));
+ }
+ return result.build();
+ }
+
/**
* Copy the given {@link MailboxMessage} to a new mailbox and return the uid of the copy. Be aware that the given uid is just a suggestion for the uid of the copied
diff --git a/mailbox/store/src/test/java/org/apache/james/mailbox/store/mail/model/MessageMapperTest.java b/mailbox/store/src/test/java/org/apache/james/mailbox/store/mail/model/MessageMapperTest.java
index 7f0d1ed..1d44644 100644
--- a/mailbox/store/src/test/java/org/apache/james/mailbox/store/mail/model/MessageMapperTest.java
+++ b/mailbox/store/src/test/java/org/apache/james/mailbox/store/mail/model/MessageMapperTest.java
@@ -319,7 +319,38 @@ public abstract class MessageMapperTest {
messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message2.getUid()));
messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message4.getUid()));
messageMapper.updateFlags(benwaWorkMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message6.getUid()));
- assertThat(messageMapper.findRecentMessageUidsInMailbox(benwaInboxMailbox)).containsOnly(message2.getUid(), message4.getUid());
+ assertThat(messageMapper.findRecentMessageUidsInMailbox(benwaInboxMailbox))
+ .containsOnly(message2.getUid(), message4.getUid());
+ }
+
+ @Test
+ void resetRecentsShouldReturnEmptyListWhenNoMessagesMarkedAsRecentArePresentInMailbox() throws MailboxException {
+ assertThat(messageMapper.resetRecent(benwaInboxMailbox)).isEmpty();
+ }
+
+ @Test
+ void resetRecentsShouldRemoveAllRecentFlags() throws MailboxException {
+ saveMessages();
+ messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message2.getUid()));
+ messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message4.getUid()));
+ messageMapper.updateFlags(benwaWorkMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message6.getUid()));
+
+ messageMapper.resetRecent(benwaInboxMailbox);
+
+ assertThat(messageMapper.findRecentMessageUidsInMailbox(benwaInboxMailbox))
+ .isEmpty();
+ }
+
+ @Test
+ void resetRecentsShouldReturnUpdatedFlags() throws MailboxException {
+ saveMessages();
+ messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message2.getUid()));
+ messageMapper.updateFlags(benwaInboxMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message4.getUid()));
+ messageMapper.updateFlags(benwaWorkMailbox, new FlagsUpdateCalculator(new Flags(Flags.Flag.RECENT), FlagsUpdateMode.REPLACE), MessageRange.one(message6.getUid()));
+
+ assertThat(messageMapper.resetRecent(benwaInboxMailbox))
+ .extracting(UpdatedFlags::getUid)
+ .containsOnly(message2.getUid(), message4.getUid());
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 03/10: JAMES-3202 Add reindexing mode in
RunningOptions
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1727a0e62f203991043493a89316cb2f1b269ae1
Author: Rene Cordier <rc...@linagora.com>
AuthorDate: Mon Jun 8 13:28:30 2020 +0700
JAMES-3202 Add reindexing mode in RunningOptions
---
.../apache/james/mailbox/indexer/ReIndexer.java | 81 +++++++++++++++++++++-
.../mailbox/tools/indexer/RunningOptionsDTO.java | 18 ++++-
...rorRecoveryIndexationTaskSerializationTest.java | 17 ++++-
.../FullReindexingTaskSerializationTest.java | 9 ++-
...ngleMailboxReindexingTaskSerializationTest.java | 9 ++-
.../UserReindexingTaskSerializationTest.java | 9 ++-
.../routes/ReindexingRunningOptionsParser.java | 11 ++-
7 files changed, 135 insertions(+), 19 deletions(-)
diff --git a/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java b/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
index 2a015f7..60c08a4 100644
--- a/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
+++ b/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
@@ -19,6 +19,9 @@
package org.apache.james.mailbox.indexer;
+import java.util.Optional;
+import java.util.stream.Stream;
+
import org.apache.james.core.Username;
import org.apache.james.mailbox.MessageUid;
import org.apache.james.mailbox.exception.MailboxException;
@@ -29,17 +32,91 @@ import org.apache.james.task.Task;
public interface ReIndexer {
class RunningOptions {
- public static final RunningOptions DEFAULT = new RunningOptions(50);
+ public static class Builder {
+ private Optional<Integer> messagesPerSecond;
+ private Optional<Mode> mode;
+
+ public Builder() {
+ this.messagesPerSecond = Optional.empty();
+ this.mode = Optional.empty();
+ }
+
+ public Builder messagesPerSeconds(Optional<Integer> messagesPerSecond) {
+ this.messagesPerSecond = messagesPerSecond;
+ return this;
+ }
+
+ public Builder messagesPerSeconds(int messagesPerSecond) {
+ return messagesPerSeconds(Optional.of(messagesPerSecond));
+ }
+
+ public Builder mode(Optional<Mode> mode) {
+ this.mode = mode;
+ return this;
+ }
+
+ public Builder mode(Mode mode) {
+ return mode(Optional.of(mode));
+ }
+
+ public RunningOptions build() {
+ return new RunningOptions(
+ messagesPerSecond.orElse(DEFAULT_MESSAGES_PER_SECONDS),
+ mode.orElse(DEFAULT_MODE)
+ );
+ }
+ }
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ public enum Mode {
+ REBUILD_ALL("rebuildAll"),
+ FIX_OUTDATED("fixOutdated");
+
+ private final String value;
+
+ Mode(String value) {
+ this.value = value;
+ }
+
+ String getValue() {
+ return value;
+ }
+
+ static Optional<Mode> fromString(String optionalMode) {
+ return Stream.of(values())
+ .filter(mode -> mode.getValue().equalsIgnoreCase(optionalMode))
+ .findFirst();
+ }
+ }
+
+ public static Optional<Mode> parseMode(String optionalMode) {
+ return Optional.ofNullable(optionalMode)
+ .flatMap(Mode::fromString);
+ }
+
+ private static final Mode DEFAULT_MODE = Mode.REBUILD_ALL;
+ private static final int DEFAULT_MESSAGES_PER_SECONDS = 50;
+
+ public static final RunningOptions DEFAULT = builder().build();
private final int messagesPerSecond;
+ private final Mode mode;
- public RunningOptions(int messagesPerSecond) {
+ private RunningOptions(int messagesPerSecond, Mode mode) {
this.messagesPerSecond = messagesPerSecond;
+ this.mode = mode;
}
public int getMessagesPerSecond() {
return messagesPerSecond;
}
+
+ public Mode getMode() {
+ return mode;
+ }
}
Task reIndex(Username username, RunningOptions runningOptions) throws MailboxException;
diff --git a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/RunningOptionsDTO.java b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/RunningOptionsDTO.java
index 16f3d5f..8ae6d56 100644
--- a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/RunningOptionsDTO.java
+++ b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/RunningOptionsDTO.java
@@ -28,22 +28,34 @@ import com.fasterxml.jackson.annotation.JsonProperty;
public class RunningOptionsDTO {
public static RunningOptionsDTO toDTO(RunningOptions runningOptions) {
- return new RunningOptionsDTO(Optional.of(runningOptions.getMessagesPerSecond()));
+ return new RunningOptionsDTO(
+ Optional.of(runningOptions.getMessagesPerSecond()),
+ Optional.of(runningOptions.getMode()));
}
private final Optional<Integer> messagesPerSecond;
+ private final Optional<RunningOptions.Mode> mode;
@JsonCreator
- public RunningOptionsDTO(@JsonProperty("messagesPerSecond") Optional<Integer> messagesPerSecond) {
+ public RunningOptionsDTO(@JsonProperty("messagesPerSecond") Optional<Integer> messagesPerSecond,
+ @JsonProperty("mode") Optional<RunningOptions.Mode> mode) {
this.messagesPerSecond = messagesPerSecond;
+ this.mode = mode;
}
public Optional<Integer> getMessagesPerSecond() {
return messagesPerSecond;
}
+ public Optional<RunningOptions.Mode> getMode() {
+ return mode;
+ }
+
public RunningOptions toDomainObject() {
- return new RunningOptions(messagesPerSecond.orElse(RunningOptions.DEFAULT.getMessagesPerSecond()));
+ return RunningOptions.builder()
+ .messagesPerSeconds(messagesPerSecond)
+ .mode(mode)
+ .build();
}
}
diff --git a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/ErrorRecoveryIndexationTaskSerializationTest.java b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/ErrorRecoveryIndexationTaskSerializationTest.java
index 3ad7d4f..7ef471c 100644
--- a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/ErrorRecoveryIndexationTaskSerializationTest.java
+++ b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/ErrorRecoveryIndexationTaskSerializationTest.java
@@ -43,10 +43,11 @@ class ErrorRecoveryIndexationTaskSerializationTest {
private final int successfullyReprocessedMailCount = 42;
private final int failedReprocessedMailCount = 2;
private final String serializedErrorRecoveryReindexingTask = "{\"type\": \"error-recovery-indexation\"," +
- " \"previousMessageFailures\" : [{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"previousMailboxFailures\": [\"3\"], \"runningOptions\":{\"messagesPerSecond\":50}}";
+ " \"previousMessageFailures\" : [{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"previousMailboxFailures\": [\"3\"], \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"REBUILD_ALL\"}}";
private final String legacySerializedErrorRecoveryReindexingTask = "{\"type\": \"error-recovery-indexation\"," +
" \"previousFailures\" : [{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}]}";
- private final String serializedAdditionalInformation = "{\"type\": \"error-recovery-indexation\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"mailboxFailures\":[\"3\"], \"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedAdditionalInformation = "{\"type\": \"error-recovery-indexation\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"mailboxFailures\":[\"3\"], \"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"REBUILD_ALL\"}}";
+ private final String serializedAdditionalInformationWithCorrectMode = "{\"type\": \"error-recovery-indexation\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"mailboxFailures\":[\"3\"], \"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"FIX_OUTDATED\"}}";
private final String legacySerializedAdditionalInformation = "{\"type\": \"error-recovery-indexation\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"failures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"timestamp\":\"2018-11-13T12:00:55Z\"}";
private final TestId mailboxId = TestId.of(1L);
private final MessageUid messageUid = MessageUid.of(10L);
@@ -97,6 +98,18 @@ class ErrorRecoveryIndexationTaskSerializationTest {
}
@Test
+ void additionalInformationWithCorrectModeShouldBeSerializable() throws Exception {
+ RunningOptions runningOptions = RunningOptions.builder()
+ .mode(RunningOptions.Mode.FIX_OUTDATED)
+ .build();
+ ReprocessingContextInformationForErrorRecoveryIndexationTask details = new ReprocessingContextInformationForErrorRecoveryIndexationTask(successfullyReprocessedMailCount, failedReprocessedMailCount, executionFailures, TIMESTAMP, runningOptions);
+ JsonSerializationVerifier.dtoModule(ReprocessingContextInformationForErrorRecoveryIndexationTask.module(mailboxIdFactory))
+ .bean(details)
+ .json(serializedAdditionalInformationWithCorrectMode)
+ .verify();
+ }
+
+ @Test
void legacyAdditionalInformationShouldBeDeserializable() throws Exception {
ReprocessingContextInformationForErrorRecoveryIndexationTask legacyAdditionalInformation = JsonGenericSerializer.forModules(ReprocessingContextInformationDTO.ReprocessingContextInformationForErrorRecoveryIndexationTask.module(new TestId.Factory()))
.withoutNestedType()
diff --git a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/FullReindexingTaskSerializationTest.java b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/FullReindexingTaskSerializationTest.java
index a19d8d1..81ae782 100644
--- a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/FullReindexingTaskSerializationTest.java
+++ b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/FullReindexingTaskSerializationTest.java
@@ -47,10 +47,10 @@ class FullReindexingTaskSerializationTest {
private final int successfullyReprocessedMailCount = 42;
private final int failedReprocessedMailCount = 2;
- private final String serializedFullReindexingTask = "{\"type\":\"full-reindexing\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedFullReindexingTask = "{\"type\":\"full-reindexing\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"REBUILD_ALL\"}}";
private final String legacySerializedFullReindexingTask = "{\"type\":\"full-reindexing\"}";
- private final String serializedAdditionalInformation = "{\"type\": \"full-reindexing\", \"runningOptions\":{\"messagesPerSecond\":50}, \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"mailboxFailures\": [\"3\"],\"timestamp\":\"2018-11-13T12:00:55Z\"}";
+ private final String serializedAdditionalInformation = "{\"type\": \"full-reindexing\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"FIX_OUTDATED\"}, \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"mailboxFailures\": [\"3\"],\"timestamp\":\"2018-11-13T12:00:55Z\"}";
private final String legacySerializedAdditionalInformation = "{\"type\": \"full-reindexing\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"failures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"timestamp\":\"2018-11-13T12:00:55Z\"}";
private ReIndexingExecutionFailures reIndexingExecutionFailures;
@@ -88,7 +88,10 @@ class FullReindexingTaskSerializationTest {
@Test
void additionalInformationShouldBeSerializable() throws Exception {
- ReprocessingContextInformationForFullReindexingTask details = new ReprocessingContextInformationForFullReindexingTask(successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, RunningOptions.DEFAULT);
+ RunningOptions runningOptions = RunningOptions.builder()
+ .mode(RunningOptions.Mode.FIX_OUTDATED)
+ .build();
+ ReprocessingContextInformationForFullReindexingTask details = new ReprocessingContextInformationForFullReindexingTask(successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, runningOptions);
JsonSerializationVerifier.dtoModule(ReprocessingContextInformationForFullReindexingTask.module(new TestId.Factory()))
.bean(details)
diff --git a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/SingleMailboxReindexingTaskSerializationTest.java b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/SingleMailboxReindexingTaskSerializationTest.java
index df5a50d..dc1eaf6 100644
--- a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/SingleMailboxReindexingTaskSerializationTest.java
+++ b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/SingleMailboxReindexingTaskSerializationTest.java
@@ -40,9 +40,9 @@ class SingleMailboxReindexingTaskSerializationTest {
private final int successfullyReprocessedMailCount = 42;
private final int failedReprocessedMailCount = 2;
- private final String serializedMailboxReindexingTask = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedMailboxReindexingTask = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"REBUILD_ALL\"}}";
private final String legacySerializedMailboxReindexingTask = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\"}";
- private final String serializedAdditionalInformation = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10, 20]}], \"mailboxFailures\": [\"2\"],\"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedAdditionalInformation = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10, 20]}], \"mailboxFailures\": [\"2\"],\"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"FIX_OUTDATED\"}}";
private final String legacySerializedAdditionalInformation = "{\"type\": \"mailbox-reindexing\", \"mailboxId\": \"1\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"failures\":[{\"mailboxId\":\"1\",\"uids\":[10, 20]}], \"timestamp\":\"2018-11-13T12:00:55Z\"}";
private final TestId mailboxId = TestId.of(1L);
@@ -86,7 +86,10 @@ class SingleMailboxReindexingTaskSerializationTest {
@Test
void additionalInformationShouldBeSerializable() throws Exception {
- SingleMailboxReindexingTask.AdditionalInformation details = new SingleMailboxReindexingTask.AdditionalInformation(mailboxId, successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, RunningOptions.DEFAULT);
+ RunningOptions runningOptions = RunningOptions.builder()
+ .mode(RunningOptions.Mode.FIX_OUTDATED)
+ .build();
+ SingleMailboxReindexingTask.AdditionalInformation details = new SingleMailboxReindexingTask.AdditionalInformation(mailboxId, successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, runningOptions);
JsonSerializationVerifier.dtoModule(SingleMailboxReindexingTaskAdditionalInformationDTO.module(new TestId.Factory()))
.bean(details)
diff --git a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/UserReindexingTaskSerializationTest.java b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/UserReindexingTaskSerializationTest.java
index 7dbb40d..d1a281c 100644
--- a/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/UserReindexingTaskSerializationTest.java
+++ b/mailbox/tools/indexer/src/test/java/org/apache/mailbox/tools/indexer/UserReindexingTaskSerializationTest.java
@@ -41,9 +41,9 @@ class UserReindexingTaskSerializationTest {
private final int successfullyReprocessedMailCount = 42;
private final int failedReprocessedMailCount = 2;
- private final String serializedUserReindexingTask = "{\"type\": \"user-reindexing\", \"username\": \"foo@apache.org\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedUserReindexingTask = "{\"type\": \"user-reindexing\", \"username\": \"foo@apache.org\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"REBUILD_ALL\"}}";
private final String legacySerializedUserReindexingTask = "{\"type\": \"user-reindexing\", \"username\": \"foo@apache.org\"}";
- private final String serializedAdditionalInformation = "{\"type\": \"user-reindexing\", \"user\": \"foo@apache.org\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}],\"mailboxFailures\":[\"3\"], \"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50}}";
+ private final String serializedAdditionalInformation = "{\"type\": \"user-reindexing\", \"user\": \"foo@apache.org\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"messageFailures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}],\"mailboxFailures\":[\"3\"], \"timestamp\":\"2018-11-13T12:00:55Z\", \"runningOptions\":{\"messagesPerSecond\":50, \"mode\":\"FIX_OUTDATED\"}}";
private final String legacySerializedAdditionalInformation = "{\"type\": \"user-reindexing\", \"user\": \"foo@apache.org\", \"successfullyReprocessedMailCount\":42,\"failedReprocessedMailCount\":2,\"failures\":[{\"mailboxId\":\"1\",\"uids\":[10]},{\"mailboxId\":\"2\",\"uids\":[20]}], \"timestamp\":\"2018-11-13T12:00:55Z\"}";
private final TestId mailboxId = TestId.of(1L);
private final MessageUid messageUid = MessageUid.of(10L);
@@ -88,7 +88,10 @@ class UserReindexingTaskSerializationTest {
@Test
void additionalInformationShouldBeSerializable() throws Exception {
- UserReindexingTask.AdditionalInformation details = new UserReindexingTask.AdditionalInformation(USERNAME, successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, RunningOptions.DEFAULT);
+ RunningOptions runningOptions = RunningOptions.builder()
+ .mode(RunningOptions.Mode.FIX_OUTDATED)
+ .build();
+ UserReindexingTask.AdditionalInformation details = new UserReindexingTask.AdditionalInformation(USERNAME, successfullyReprocessedMailCount, failedReprocessedMailCount, reIndexingExecutionFailures, TIMESTAMP, runningOptions);
JsonSerializationVerifier.dtoModule(UserReindexingTaskAdditionalInformationDTO.module(new TestId.Factory()))
.bean(details)
.json(serializedAdditionalInformation)
diff --git a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/ReindexingRunningOptionsParser.java b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/ReindexingRunningOptionsParser.java
index c3f869e..d5ce613 100644
--- a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/ReindexingRunningOptionsParser.java
+++ b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/ReindexingRunningOptionsParser.java
@@ -28,9 +28,10 @@ import spark.Request;
public class ReindexingRunningOptionsParser {
public static RunningOptions parse(Request request) {
- return intQueryParameter(request, "messagesPerSecond")
- .map(RunningOptions::new)
- .orElse(RunningOptions.DEFAULT);
+ return RunningOptions.builder()
+ .messagesPerSeconds(intQueryParameter(request, "messagesPerSecond"))
+ .mode(modeQueryParameter(request, "mode"))
+ .build();
}
public static Optional<Integer> intQueryParameter(Request request, String queryParameter) {
@@ -42,4 +43,8 @@ public class ReindexingRunningOptionsParser {
"strictly positive optional integer", queryParameter), e);
}
}
+
+ public static Optional<RunningOptions.Mode> modeQueryParameter(Request request, String queryParameter) {
+ return RunningOptions.parseMode(request.queryParams(queryParameter));
+ }
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 05/10: JAMES-3202 Integration tests for Reindex
with Mode set to Correct
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit bdfdeb6c6b1eaa6dce5cfdacafb82be0786b86d9
Author: Rene Cordier <rc...@linagora.com>
AuthorDate: Fri Jun 12 17:00:04 2020 +0700
JAMES-3202 Integration tests for Reindex with Mode set to Correct
---
server/protocols/webadmin/webadmin-mailbox/pom.xml | 18 +-
.../james/webadmin/routes/MailboxesRoutesTest.java | 423 ++++++++++++++++++++-
.../webadmin/routes/UserMailboxesRoutesTest.java | 259 ++++++++++++-
3 files changed, 666 insertions(+), 34 deletions(-)
diff --git a/server/protocols/webadmin/webadmin-mailbox/pom.xml b/server/protocols/webadmin/webadmin-mailbox/pom.xml
index c4868ce..89777b6 100644
--- a/server/protocols/webadmin/webadmin-mailbox/pom.xml
+++ b/server/protocols/webadmin/webadmin-mailbox/pom.xml
@@ -35,7 +35,8 @@
<dependencies>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>james-server-dnsservice-test</artifactId>
+ <artifactId>apache-james-backends-es</artifactId>
+ <type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
@@ -50,6 +51,10 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-elasticsearch</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>apache-james-mailbox-event-json</artifactId>
</dependency>
<dependency>
@@ -93,12 +98,6 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
- <artifactId>apache-james-backends-es</artifactId>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>${james.groupId}</groupId>
<artifactId>apache-james-mailbox-quota-search-elasticsearch</artifactId>
<scope>test</scope>
</dependency>
@@ -153,6 +152,11 @@
</dependency>
<dependency>
<groupId>${james.groupId}</groupId>
+ <artifactId>james-server-dnsservice-test</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
<artifactId>james-server-task-json</artifactId>
</dependency>
<dependency>
diff --git a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
index 65ce22a..244331a 100644
--- a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
+++ b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
@@ -23,28 +23,59 @@ import static io.restassured.RestAssured.given;
import static io.restassured.RestAssured.when;
import static io.restassured.RestAssured.with;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.notNullValue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
+import java.nio.charset.StandardCharsets;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.util.Date;
+import java.util.List;
+
+import javax.mail.Flags;
+import javax.mail.util.SharedByteArrayInputStream;
+
+import org.apache.james.backends.es.DockerElasticSearchExtension;
+import org.apache.james.backends.es.ElasticSearchIndexer;
+import org.apache.james.backends.es.ReactorElasticSearchClient;
import org.apache.james.core.Username;
import org.apache.james.json.DTOConverter;
import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MessageIdManager;
import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.MessageUid;
+import org.apache.james.mailbox.elasticsearch.IndexAttachments;
+import org.apache.james.mailbox.elasticsearch.MailboxElasticSearchConstants;
+import org.apache.james.mailbox.elasticsearch.MailboxIdRoutingKeyFactory;
+import org.apache.james.mailbox.elasticsearch.MailboxIndexCreationUtil;
+import org.apache.james.mailbox.elasticsearch.events.ElasticSearchListeningMessageSearchIndex;
+import org.apache.james.mailbox.elasticsearch.json.MessageToElasticSearchJson;
+import org.apache.james.mailbox.elasticsearch.query.CriterionConverter;
+import org.apache.james.mailbox.elasticsearch.query.QueryConverter;
+import org.apache.james.mailbox.elasticsearch.search.ElasticSearchSearcher;
import org.apache.james.mailbox.indexer.ReIndexer;
import org.apache.james.mailbox.inmemory.InMemoryId;
import org.apache.james.mailbox.inmemory.InMemoryMailboxManager;
+import org.apache.james.mailbox.inmemory.InMemoryMessageId;
import org.apache.james.mailbox.inmemory.manager.InMemoryIntegrationResources;
import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.mailbox.model.UpdatedFlags;
+import org.apache.james.mailbox.store.extractor.DefaultTextExtractor;
import org.apache.james.mailbox.store.mail.model.MailboxMessage;
+import org.apache.james.mailbox.store.mail.model.impl.PropertyBuilder;
+import org.apache.james.mailbox.store.mail.model.impl.SimpleMailboxMessage;
import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
import org.apache.james.task.Hostname;
import org.apache.james.task.MemoryTaskManager;
@@ -63,35 +94,74 @@ import org.apache.mailbox.tools.indexer.SingleMailboxReindexingTask;
import org.apache.mailbox.tools.indexer.SingleMessageReindexingTask;
import org.apache.mailbox.tools.indexer.SingleMessageReindexingTaskAdditionalInformationDTO;
import org.eclipse.jetty.http.HttpStatus;
+import org.elasticsearch.index.IndexNotFoundException;
import org.hamcrest.Matchers;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
import org.mockito.ArgumentCaptor;
-import org.mockito.Mockito;
import org.testcontainers.shaded.com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableList;
+
import io.restassured.RestAssured;
import reactor.core.publisher.Mono;
class MailboxesRoutesTest {
- private static final Username USERNAME = Username.of("benwa@apache.org");
- private static final MailboxPath INBOX = MailboxPath.inbox(USERNAME);
+ static final Username USERNAME = Username.of("benwa@apache.org");
+ static final MailboxPath INBOX = MailboxPath.inbox(USERNAME);
+ static final int BATCH_SIZE = 1;
+ static final int SEARCH_SIZE = 1;
+
+ @RegisterExtension
+ DockerElasticSearchExtension elasticSearch = new DockerElasticSearchExtension();
- private WebAdminServer webAdminServer;
- private ListeningMessageSearchIndex searchIndex;
- private InMemoryMailboxManager mailboxManager;
- private MemoryTaskManager taskManager;
+ WebAdminServer webAdminServer;
+ ListeningMessageSearchIndex searchIndex;
+ InMemoryMailboxManager mailboxManager;
+ MessageIdManager messageIdManager;
+ MemoryTaskManager taskManager;
+ ReactorElasticSearchClient client;
@BeforeEach
- void beforeEach() {
- mailboxManager = InMemoryIntegrationResources.defaultResources().getMailboxManager();
+ void beforeEach() throws Exception {
+ client = MailboxIndexCreationUtil.prepareDefaultClient(
+ elasticSearch.getDockerElasticSearch().clientProvider().get(),
+ elasticSearch.getDockerElasticSearch().configuration());
+
+ InMemoryMessageId.Factory messageIdFactory = new InMemoryMessageId.Factory();
+ MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
+
+ InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
+ .preProvisionnedFakeAuthenticator()
+ .fakeAuthorizator()
+ .inVmEventBus()
+ .defaultAnnotationLimits()
+ .defaultMessageParser()
+ .listeningSearchIndex(preInstanciationStage -> new ElasticSearchListeningMessageSearchIndex(
+ preInstanciationStage.getMapperFactory(),
+ new ElasticSearchIndexer(client,
+ MailboxElasticSearchConstants.DEFAULT_MAILBOX_WRITE_ALIAS,
+ BATCH_SIZE),
+ new ElasticSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE,
+ new InMemoryId.Factory(), messageIdFactory,
+ MailboxElasticSearchConstants.DEFAULT_MAILBOX_READ_ALIAS, routingKeyFactory),
+ new MessageToElasticSearchJson(new DefaultTextExtractor(), ZoneId.of("Europe/Paris"), IndexAttachments.YES),
+ preInstanciationStage.getSessionProvider(), routingKeyFactory))
+ .noPreDeletionHooks()
+ .storeQuotaManager()
+ .build();
+
+ mailboxManager = resources.getMailboxManager();
+ messageIdManager = resources.getMessageIdManager();
taskManager = new MemoryTaskManager(new Hostname("foo"));
InMemoryId.Factory mailboxIdFactory = new InMemoryId.Factory();
- searchIndex = mock(ListeningMessageSearchIndex.class);
- Mockito.when(searchIndex.add(any(), any(), any())).thenReturn(Mono.empty());
- Mockito.when(searchIndex.deleteAll(any(), any())).thenReturn(Mono.empty());
+
+ searchIndex = spy((ListeningMessageSearchIndex) resources.getSearchIndex());
+
ReIndexerPerformer reIndexerPerformer = new ReIndexerPerformer(
mailboxManager,
searchIndex,
@@ -271,6 +341,163 @@ class MailboxesRoutesTest {
.body("taskId", Matchers.is(notNullValue()))
.body("additionalInformation.mailboxFailures", Matchers.containsInAnyOrder(mailboxId.serialize()));
}
+
+ @Test
+ void fullReprocessingWithCorrectModeShouldReturnTaskDetailsWhenMails() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+ mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession);
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=fixOutdated")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await")
+ .then()
+ .body("status", is("completed"))
+ .body("taskId", is(notNullValue()))
+ .body("type", is(FullReindexingTask.FULL_RE_INDEXING.asString()))
+ .body("additionalInformation.successfullyReprocessedMailCount", is(2))
+ .body("additionalInformation.failedReprocessedMailCount", is(0))
+ .body("startedDate", is(notNullValue()))
+ .body("submitDate", is(notNullValue()))
+ .body("completedDate", is(notNullValue()));
+ }
+
+ @Test
+ void fullReprocessingWithCorrectModeShouldFixInconsistenciesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=fixOutdated")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Test
+ void fullReprocessingWithCorrectModeShouldNotChangeDocumentsInESWhenNoInconsistencies() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=fixOutdated")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Disabled("JAMES-3202 Limitation of the current correct mode reindexation. We only check metadata and fix "
+ + "inconsistencies with ES, but we don't check for inconsistencies from ES to metadata")
+ @Test
+ void fullReprocessingWithCorrectModeShouldRemoveOrphanMessagesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ byte[] content = "Simple message content".getBytes(StandardCharsets.UTF_8);
+ MessageUid uid = MessageUid.of(22L);
+
+ SimpleMailboxMessage message = SimpleMailboxMessage.builder()
+ .messageId(InMemoryMessageId.of(42L))
+ .uid(uid)
+ .content(new SharedByteArrayInputStream(content))
+ .size(content.length)
+ .internalDate(new Date(ZonedDateTime.parse("2018-02-15T15:54:02Z").toEpochSecond()))
+ .bodyStartOctet(0)
+ .flags(new Flags("myFlags"))
+ .propertyBuilder(new PropertyBuilder())
+ .mailboxId(mailboxId)
+ .build();
+
+ searchIndex.add(systemSession, mailbox, message).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=fixOutdated")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThatThrownBy(() -> searchIndex.retrieveIndexedFlags(mailbox, uid).block())
+ .isInstanceOf(IndexNotFoundException.class);
+ }
}
@Nested
@@ -485,6 +712,172 @@ class MailboxesRoutesTest {
.body("taskId", Matchers.is(notNullValue()))
.body("additionalInformation.mailboxFailures", Matchers.containsInAnyOrder(mailboxId.serialize()));
}
+
+
+ @Test
+ void mailboxReprocessingWithCorrectModeShouldReturnTaskDetailsWhenMails() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+ mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession);
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post("/mailboxes/" + mailboxId.serialize())
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await")
+ .then()
+ .body("status", is("completed"))
+ .body("taskId", is(notNullValue()))
+ .body("type", is(SingleMailboxReindexingTask.TYPE.asString()))
+ .body("additionalInformation.successfullyReprocessedMailCount", is(2))
+ .body("additionalInformation.failedReprocessedMailCount", is(0))
+ .body("startedDate", is(notNullValue()))
+ .body("submitDate", is(notNullValue()))
+ .body("completedDate", is(notNullValue()));
+ }
+
+ @Test
+ void mailboxReprocessingWithCorrectModeShouldFixInconsistenciesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post("/mailboxes/" + mailboxId.serialize())
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Test
+ void mailboxReprocessingWithCorrectModeShouldNotChangeDocumentsInESWhenNoInconsistencies() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post("/mailboxes/" + mailboxId.serialize())
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Disabled("JAMES-3202 Limitation of the current correct mode reindexation. We only check metadata and fix "
+ + "inconsistencies with ES, but we don't check for inconsistencies from ES to metadata")
+ @Test
+ void mailboxReprocessingWithCorrectModeShouldRemoveOrphanMessagesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ byte[] content = "Simple message content".getBytes(StandardCharsets.UTF_8);
+ MessageUid uid = MessageUid.of(22L);
+
+ SimpleMailboxMessage message = SimpleMailboxMessage.builder()
+ .messageId(InMemoryMessageId.of(42L))
+ .uid(uid)
+ .content(new SharedByteArrayInputStream(content))
+ .size(content.length)
+ .internalDate(new Date(ZonedDateTime.parse("2018-02-15T15:54:02Z").toEpochSecond()))
+ .bodyStartOctet(0)
+ .flags(new Flags("myFlags"))
+ .propertyBuilder(new PropertyBuilder())
+ .mailboxId(mailboxId)
+ .build();
+
+ searchIndex.add(systemSession, mailbox, message).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post("/mailboxes/" + mailboxId.serialize())
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThatThrownBy(() -> searchIndex.retrieveIndexedFlags(mailbox, uid).block())
+ .isInstanceOf(IndexNotFoundException.class);
+ }
}
@Nested
@@ -838,7 +1231,7 @@ class MailboxesRoutesTest {
}
@Test
- void mailboxReprocessingShouldReturnTaskDetailsWhenFailing() throws Exception {
+ void fixingReIndexingShouldReturnTaskDetailsWhenFailing() throws Exception {
MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
ComposedMessageId composedMessageId = mailboxManager.getMailbox(INBOX, systemSession)
@@ -937,8 +1330,6 @@ class MailboxesRoutesTest {
.get(taskId + "/await");
reset(searchIndex);
- Mockito.when(searchIndex.add(any(), any(), any())).thenReturn(Mono.empty());
- Mockito.when(searchIndex.deleteAll(any(), any())).thenReturn(Mono.empty());
String fixingTaskId = with()
.queryParam("reIndexFailedMessagesOf", taskId)
diff --git a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/UserMailboxesRoutesTest.java b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/UserMailboxesRoutesTest.java
index ff3dbfb..3e18b00 100644
--- a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/UserMailboxesRoutesTest.java
+++ b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/UserMailboxesRoutesTest.java
@@ -26,6 +26,7 @@ import static io.restassured.http.ContentType.JSON;
import static org.apache.james.webadmin.Constants.SEPARATOR;
import static org.apache.james.webadmin.routes.UserMailboxesRoutes.USERS_BASE;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.eclipse.jetty.http.HttpStatus.BAD_REQUEST_400;
import static org.eclipse.jetty.http.HttpStatus.INTERNAL_SERVER_ERROR_500;
import static org.eclipse.jetty.http.HttpStatus.NOT_FOUND_404;
@@ -38,35 +39,63 @@ import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
+import java.nio.charset.StandardCharsets;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.util.Date;
import java.util.List;
import java.util.Map;
-import java.util.stream.Stream;
+import javax.mail.Flags;
+import javax.mail.util.SharedByteArrayInputStream;
+
+import org.apache.james.backends.es.DockerElasticSearchExtension;
+import org.apache.james.backends.es.ElasticSearchIndexer;
+import org.apache.james.backends.es.ReactorElasticSearchClient;
import org.apache.james.core.Username;
import org.apache.james.json.DTOConverter;
import org.apache.james.mailbox.MailboxManager;
import org.apache.james.mailbox.MailboxSession;
import org.apache.james.mailbox.MailboxSessionUtil;
+import org.apache.james.mailbox.MessageIdManager;
import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.MessageUid;
+import org.apache.james.mailbox.elasticsearch.IndexAttachments;
+import org.apache.james.mailbox.elasticsearch.MailboxElasticSearchConstants;
+import org.apache.james.mailbox.elasticsearch.MailboxIdRoutingKeyFactory;
+import org.apache.james.mailbox.elasticsearch.MailboxIndexCreationUtil;
+import org.apache.james.mailbox.elasticsearch.events.ElasticSearchListeningMessageSearchIndex;
+import org.apache.james.mailbox.elasticsearch.json.MessageToElasticSearchJson;
+import org.apache.james.mailbox.elasticsearch.query.CriterionConverter;
+import org.apache.james.mailbox.elasticsearch.query.QueryConverter;
+import org.apache.james.mailbox.elasticsearch.search.ElasticSearchSearcher;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.MailboxExistsException;
import org.apache.james.mailbox.exception.MailboxNotFoundException;
import org.apache.james.mailbox.indexer.ReIndexer;
import org.apache.james.mailbox.inmemory.InMemoryId;
import org.apache.james.mailbox.inmemory.InMemoryMailboxManager;
+import org.apache.james.mailbox.inmemory.InMemoryMessageId;
import org.apache.james.mailbox.inmemory.manager.InMemoryIntegrationResources;
import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MailboxMetaData;
import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.mailbox.model.UpdatedFlags;
import org.apache.james.mailbox.model.search.MailboxQuery;
import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.extractor.DefaultTextExtractor;
import org.apache.james.mailbox.store.mail.model.MailboxMessage;
+import org.apache.james.mailbox.store.mail.model.impl.PropertyBuilder;
+import org.apache.james.mailbox.store.mail.model.impl.SimpleMailboxMessage;
import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
import org.apache.james.task.Hostname;
import org.apache.james.task.MemoryTaskManager;
@@ -79,14 +108,18 @@ import org.apache.james.webadmin.utils.JsonTransformer;
import org.apache.mailbox.tools.indexer.ReIndexerImpl;
import org.apache.mailbox.tools.indexer.ReIndexerPerformer;
import org.apache.mailbox.tools.indexer.UserReindexingTask;
+import org.elasticsearch.index.IndexNotFoundException;
import org.hamcrest.Matchers;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
import org.mockito.ArgumentCaptor;
import org.mockito.Mockito;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import io.restassured.RestAssured;
@@ -103,18 +136,13 @@ class UserMailboxesRoutesTest {
private WebAdminServer webAdminServer;
private UsersRepository usersRepository;
- private ListeningMessageSearchIndex searchIndex;
private MemoryTaskManager taskManager;
- private void createServer(MailboxManager mailboxManager, MailboxSessionMapperFactory mapperFactory, MailboxId.Factory mailboxIdFactory) throws Exception {
+ private void createServer(MailboxManager mailboxManager, MailboxSessionMapperFactory mapperFactory, MailboxId.Factory mailboxIdFactory, ListeningMessageSearchIndex searchIndex) throws Exception {
usersRepository = mock(UsersRepository.class);
when(usersRepository.contains(USERNAME)).thenReturn(true);
-
taskManager = new MemoryTaskManager(new Hostname("foo"));
- searchIndex = mock(ListeningMessageSearchIndex.class);
- Mockito.when(searchIndex.add(any(), any(), any())).thenReturn(Mono.empty());
- Mockito.when(searchIndex.deleteAll(any(), any())).thenReturn(Mono.empty());
ReIndexerPerformer reIndexerPerformer = new ReIndexerPerformer(
mailboxManager,
searchIndex,
@@ -149,7 +177,11 @@ class UserMailboxesRoutesTest {
@BeforeEach
void setUp() throws Exception {
InMemoryMailboxManager mailboxManager = InMemoryIntegrationResources.defaultResources().getMailboxManager();
- createServer(mailboxManager, mailboxManager.getMapperFactory(), new InMemoryId.Factory());
+ ListeningMessageSearchIndex searchIndex = mock(ListeningMessageSearchIndex.class);
+ Mockito.when(searchIndex.add(any(), any(), any())).thenReturn(Mono.empty());
+ Mockito.when(searchIndex.deleteAll(any(), any())).thenReturn(Mono.empty());
+
+ createServer(mailboxManager, mailboxManager.getMapperFactory(), new InMemoryId.Factory(), searchIndex);
}
@Test
@@ -857,8 +889,11 @@ class UserMailboxesRoutesTest {
void setUp() throws Exception {
mailboxManager = mock(MailboxManager.class);
when(mailboxManager.createSystemSession(any())).thenReturn(MailboxSessionUtil.create(USERNAME));
+ ListeningMessageSearchIndex searchIndex = mock(ListeningMessageSearchIndex.class);
+ Mockito.when(searchIndex.add(any(), any(), any())).thenReturn(Mono.empty());
+ Mockito.when(searchIndex.deleteAll(any(), any())).thenReturn(Mono.empty());
- createServer(mailboxManager, mock(MailboxSessionMapperFactory.class), new InMemoryId.Factory());
+ createServer(mailboxManager, mock(MailboxSessionMapperFactory.class), new InMemoryId.Factory(), searchIndex);
}
@Test
@@ -1114,13 +1149,50 @@ class UserMailboxesRoutesTest {
@Nested
class UserReprocessing {
+ static final int BATCH_SIZE = 1;
+ static final int SEARCH_SIZE = 1;
+
+ @RegisterExtension
+ DockerElasticSearchExtension elasticSearch = new DockerElasticSearchExtension();
private InMemoryMailboxManager mailboxManager;
+ private ListeningMessageSearchIndex searchIndex;
+ MessageIdManager messageIdManager;
@BeforeEach
void setUp() throws Exception {
- mailboxManager = InMemoryIntegrationResources.defaultResources().getMailboxManager();
- createServer(mailboxManager, mailboxManager.getMapperFactory(), new InMemoryId.Factory());
+ ReactorElasticSearchClient client = MailboxIndexCreationUtil.prepareDefaultClient(
+ elasticSearch.getDockerElasticSearch().clientProvider().get(),
+ elasticSearch.getDockerElasticSearch().configuration());
+
+ InMemoryMessageId.Factory messageIdFactory = new InMemoryMessageId.Factory();
+ MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
+
+ InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
+ .preProvisionnedFakeAuthenticator()
+ .fakeAuthorizator()
+ .inVmEventBus()
+ .defaultAnnotationLimits()
+ .defaultMessageParser()
+ .listeningSearchIndex(preInstanciationStage -> new ElasticSearchListeningMessageSearchIndex(
+ preInstanciationStage.getMapperFactory(),
+ new ElasticSearchIndexer(client,
+ MailboxElasticSearchConstants.DEFAULT_MAILBOX_WRITE_ALIAS,
+ BATCH_SIZE),
+ new ElasticSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE,
+ new InMemoryId.Factory(), messageIdFactory,
+ MailboxElasticSearchConstants.DEFAULT_MAILBOX_READ_ALIAS, routingKeyFactory),
+ new MessageToElasticSearchJson(new DefaultTextExtractor(), ZoneId.of("Europe/Paris"), IndexAttachments.YES),
+ preInstanciationStage.getSessionProvider(), routingKeyFactory))
+ .noPreDeletionHooks()
+ .storeQuotaManager()
+ .build();
+
+ mailboxManager = resources.getMailboxManager();
+ messageIdManager = resources.getMessageIdManager();
+ searchIndex = spy((ListeningMessageSearchIndex) resources.getSearchIndex());
+
+ createServer(mailboxManager, mailboxManager.getMapperFactory(), new InMemoryId.Factory(), searchIndex);
}
@Nested
@@ -1289,6 +1361,171 @@ class UserMailboxesRoutesTest {
.body("taskId", Matchers.is(notNullValue()))
.body("additionalInformation.mailboxFailures", Matchers.containsInAnyOrder(mailboxId.serialize()));
}
+
+ @Test
+ void userReprocessingWithCorrectModeShouldReturnTaskDetailsWhenMails() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+ mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession);
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post()
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await")
+ .then()
+ .body("status", Matchers.is("completed"))
+ .body("taskId", Matchers.is(notNullValue()))
+ .body("type", Matchers.is(UserReindexingTask.USER_RE_INDEXING.asString()))
+ .body("additionalInformation.successfullyReprocessedMailCount", Matchers.is(2))
+ .body("additionalInformation.failedReprocessedMailCount", Matchers.is(0))
+ .body("startedDate", Matchers.is(notNullValue()))
+ .body("submitDate", Matchers.is(notNullValue()))
+ .body("completedDate", Matchers.is(notNullValue()));
+ }
+
+ @Test
+ void userReprocessingWithCorrectModeShouldFixInconsistenciesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox, ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post()
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Test
+ void userReprocessingWithCorrectModeShouldNotChangeDocumentsInESWhenNoInconsistencies() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post()
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Disabled("JAMES-3202 Limitation of the current correct mode reindexation. We only check metadata and fix "
+ + "inconsistencies with ES, but we don't check for inconsistencies from ES to metadata")
+ @Test
+ void userReprocessingWithCorrectModeShouldRemoveOrphanMessagesInES() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ byte[] content = "Simple message content".getBytes(StandardCharsets.UTF_8);
+ MessageUid uid = MessageUid.of(22L);
+
+ SimpleMailboxMessage message = SimpleMailboxMessage.builder()
+ .messageId(InMemoryMessageId.of(42L))
+ .uid(uid)
+ .content(new SharedByteArrayInputStream(content))
+ .size(content.length)
+ .internalDate(new Date(ZonedDateTime.parse("2018-02-15T15:54:02Z").toEpochSecond()))
+ .bodyStartOctet(0)
+ .flags(new Flags("myFlags"))
+ .propertyBuilder(new PropertyBuilder())
+ .mailboxId(mailboxId)
+ .build();
+
+ searchIndex.add(systemSession, mailbox, message).block();
+
+ String taskId = with()
+ .queryParam("task", "reIndex")
+ .queryParam("mode", "fixOutdated")
+ .post()
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThatThrownBy(() -> searchIndex.retrieveIndexedFlags(mailbox, uid).block())
+ .isInstanceOf(IndexNotFoundException.class);
+ }
}
@Nested
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 04/10: JAMES-3202 Reindex only outdated documents
with the Mode option set to CORRECT in reindexing tasks
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 992cec416423cd5a71bfc6d07b33e6bb304fe58c
Author: Rene Cordier <rc...@linagora.com>
AuthorDate: Thu Jun 11 17:47:08 2020 +0700
JAMES-3202 Reindex only outdated documents with the Mode option set to CORRECT in reindexing tasks
---
.../mailbox/tools/indexer/ReIndexerPerformer.java | 108 ++++++++++++++-------
1 file changed, 71 insertions(+), 37 deletions(-)
diff --git a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
index 34d1a95..b854d67 100644
--- a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
+++ b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
@@ -19,11 +19,10 @@
package org.apache.mailbox.tools.indexer;
-import static org.apache.james.mailbox.store.mail.AbstractMessageMapper.UNLIMITED;
-
import java.time.Duration;
import javax.inject.Inject;
+import javax.mail.Flags;
import org.apache.james.core.Username;
import org.apache.james.mailbox.MailboxManager;
@@ -56,24 +55,25 @@ import reactor.core.publisher.Mono;
public class ReIndexerPerformer {
public static final int MAILBOX_CONCURRENCY = 1;
+ public static final int ONE = 1;
private static class ReIndexingEntry {
private final Mailbox mailbox;
private final MailboxSession mailboxSession;
- private final MailboxMessage message;
+ private final MessageUid uid;
- ReIndexingEntry(Mailbox mailbox, MailboxSession mailboxSession, MailboxMessage message) {
+ ReIndexingEntry(Mailbox mailbox, MailboxSession mailboxSession, MessageUid uid) {
this.mailbox = mailbox;
this.mailboxSession = mailboxSession;
- this.message = message;
+ this.uid = uid;
}
public Mailbox getMailbox() {
return mailbox;
}
- public MailboxMessage getMessage() {
- return message;
+ public MessageUid getUid() {
+ return uid;
}
public MailboxSession getMailboxSession() {
@@ -149,7 +149,7 @@ public class ReIndexerPerformer {
LOGGER.info("Starting a full reindex");
Flux<Either<Failure, ReIndexingEntry>> entriesToIndex = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).list()
- .flatMap(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession), MAILBOX_CONCURRENCY);
+ .flatMap(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession, runningOptions), MAILBOX_CONCURRENCY);
return reIndexMessages(entriesToIndex, runningOptions, reprocessingContext)
.doFinally(any -> LOGGER.info("Full reindex finished"));
@@ -160,7 +160,7 @@ public class ReIndexerPerformer {
Flux<Either<Failure, ReIndexingEntry>> entriesToIndex = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession)
.findMailboxById(mailboxId)
- .flatMapMany(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession));
+ .flatMapMany(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession, runningOptions));
return reIndexMessages(entriesToIndex, runningOptions, reprocessingContext);
}
@@ -174,7 +174,7 @@ public class ReIndexerPerformer {
try {
Flux<Either<Failure, ReIndexingEntry>> entriesToIndex = mailboxMapper.findMailboxWithPathLike(mailboxQuery.asUserBound())
- .flatMap(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession), MAILBOX_CONCURRENCY);
+ .flatMap(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession, runningOptions), MAILBOX_CONCURRENCY);
return reIndexMessages(entriesToIndex, runningOptions, reprocessingContext)
.doFinally(any -> LOGGER.info("User {} reindex finished", username.asString()));
@@ -189,9 +189,9 @@ public class ReIndexerPerformer {
return mailboxSessionMapperFactory.getMailboxMapper(mailboxSession)
.findMailboxById(mailboxId)
- .flatMap(mailbox -> fullyReadMessage(mailboxSession, mailbox, uid)
- .map(message -> Either.<Failure, ReIndexingEntry>right(new ReIndexingEntry(mailbox, mailboxSession, message)))
- .flatMap(entryOrFailure -> reIndex(entryOrFailure, reprocessingContext)))
+ .map(mailbox -> new ReIndexingEntry(mailbox, mailboxSession, uid))
+ .flatMap(this::fullyReadMessage)
+ .flatMap(message -> reIndex(message, mailboxSession))
.switchIfEmpty(Mono.just(Result.COMPLETED));
}
@@ -218,11 +218,11 @@ public class ReIndexerPerformer {
.flatMap(this::createReindexingEntryFromFailure),
Flux.fromIterable(previousReIndexingFailures.mailboxFailures())
.flatMap(mailboxId -> mapper.findMailboxById(mailboxId)
- .flatMapMany(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession))
+ .flatMapMany(mailbox -> reIndexingEntriesForMailbox(mailbox, mailboxSession, runningOptions))
.onErrorResume(e -> {
LOGGER.warn("Failed to re-index {}", mailboxId, e);
return Mono.just(Either.left(new MailboxFailure(mailboxId)));
- })));
+ }), MAILBOX_CONCURRENCY));
return reIndexMessages(entriesToIndex, runningOptions, reprocessingContext);
}
@@ -238,9 +238,9 @@ public class ReIndexerPerformer {
});
}
- private Mono<MailboxMessage> fullyReadMessage(MailboxSession mailboxSession, Mailbox mailbox, MessageUid mUid) {
- return mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
- .findInMailboxReactive(mailbox, MessageRange.one(mUid), MessageMapper.FetchType.Full, SINGLE_MESSAGE)
+ private Mono<MailboxMessage> fullyReadMessage(ReIndexingEntry entry) {
+ return mailboxSessionMapperFactory.getMessageMapper(entry.getMailboxSession())
+ .findInMailboxReactive(entry.getMailbox(), MessageRange.one(entry.getUid()), MessageMapper.FetchType.Full, SINGLE_MESSAGE)
.next();
}
@@ -249,46 +249,44 @@ public class ReIndexerPerformer {
return mailboxSessionMapperFactory.getMailboxMapper(mailboxSession)
.findMailboxById(previousFailure.getMailboxId())
- .flatMap(mailbox -> fullyReadMessage(mailboxSession, mailbox, previousFailure.getUid())
- .map(message -> Either.<Failure, ReIndexingEntry>right(new ReIndexingEntry(mailbox, mailboxSession, message))))
+ .map(mailbox -> Either.<Failure, ReIndexingEntry>right(new ReIndexingEntry(mailbox, mailboxSession, previousFailure.getUid())))
.onErrorResume(e -> {
LOGGER.warn("ReIndexing failed for {}", previousFailure, e);
return Mono.just(Either.left(new MessageFailure(previousFailure.getMailboxId(), previousFailure.getUid())));
});
}
- private Flux<Either<Failure, ReIndexingEntry>> reIndexingEntriesForMailbox(Mailbox mailbox, MailboxSession mailboxSession) {
+ private Flux<Either<Failure, ReIndexingEntry>> reIndexingEntriesForMailbox(Mailbox mailbox, MailboxSession mailboxSession, RunningOptions runningOptions) {
MessageMapper messageMapper = mailboxSessionMapperFactory.getMessageMapper(mailboxSession);
- return messageSearchIndex.deleteAll(mailboxSession, mailbox.getMailboxId())
+ return updateSearchIndex(mailbox, mailboxSession, runningOptions)
.thenMany(messageMapper.listAllMessageUids(mailbox))
- .flatMap(uid -> reIndexingEntryForUid(mailbox, mailboxSession, messageMapper, uid))
+ .map(uid -> Either.<Failure, ReIndexingEntry>right(new ReIndexingEntry(mailbox, mailboxSession, uid)))
.onErrorResume(e -> {
LOGGER.warn("ReIndexing failed for {}", mailbox.generateAssociatedPath(), e);
return Mono.just(Either.left(new MailboxFailure(mailbox.getMailboxId())));
});
}
- private Flux<Either<Failure, ReIndexingEntry>> reIndexingEntryForUid(Mailbox mailbox, MailboxSession mailboxSession, MessageMapper messageMapper, MessageUid uid) {
- return messageMapper.findInMailboxReactive(mailbox, MessageRange.one(uid), MessageMapper.FetchType.Full, UNLIMITED)
- .map(message -> Either.<Failure, ReIndexingEntry>right(new ReIndexingEntry(mailbox, mailboxSession, message)))
- .onErrorResume(e -> {
- LOGGER.warn("ReIndexing failed for {} {}", mailbox.getMailboxId(), uid, e);
- return Mono.just(Either.left(new MessageFailure(mailbox.getMailboxId(), uid)));
- });
+ private Mono<Void> updateSearchIndex(Mailbox mailbox, MailboxSession mailboxSession, RunningOptions runningOptions) {
+ if (runningOptions.getMode() == RunningOptions.Mode.REBUILD_ALL) {
+ return messageSearchIndex.deleteAll(mailboxSession, mailbox.getMailboxId());
+ }
+ return Mono.empty();
}
private Mono<Task.Result> reIndexMessages(Flux<Either<Failure, ReIndexingEntry>> entriesToIndex, RunningOptions runningOptions, ReprocessingContext reprocessingContext) {
- return entriesToIndex.transform(ReactorUtils.<Either<Failure, ReIndexingEntry>, Task.Result>throttle()
+ return entriesToIndex.transform(
+ ReactorUtils.<Either<Failure, ReIndexingEntry>, Task.Result>throttle()
.elements(runningOptions.getMessagesPerSecond())
.per(Duration.ofSeconds(1))
- .forOperation(entry -> reIndex(entry, reprocessingContext)))
+ .forOperation(entry -> reIndex(entry, reprocessingContext, runningOptions)))
.reduce(Task::combine)
.switchIfEmpty(Mono.just(Result.COMPLETED));
}
- private Mono<Task.Result> reIndex(Either<Failure, ReIndexingEntry> failureOrEntry, ReprocessingContext reprocessingContext) {
- return toMono(failureOrEntry.map(this::index))
+ private Mono<Task.Result> reIndex(Either<Failure, ReIndexingEntry> failureOrEntry, ReprocessingContext reprocessingContext, RunningOptions runningOptions) {
+ return toMono(failureOrEntry.map(entry -> reIndex(entry, runningOptions)))
.map(this::flatten)
.map(failureOrTaskResult -> recordIndexingResult(failureOrTaskResult, reprocessingContext));
}
@@ -302,15 +300,51 @@ public class ReIndexerPerformer {
result -> result.onComplete(reprocessingContext::recordSuccess));
}
+ private Mono<Either<Failure, Result>> reIndex(ReIndexingEntry entry, RunningOptions runningOptions) {
+ if (runningOptions.getMode() == RunningOptions.Mode.FIX_OUTDATED) {
+ return correctIfNeeded(entry);
+ }
+ return index(entry);
+ }
+
private Mono<Either<Failure, Result>> index(ReIndexingEntry entry) {
- return messageSearchIndex.add(entry.getMailboxSession(), entry.getMailbox(), entry.getMessage())
+ return fullyReadMessage(entry)
+ .flatMap(message -> messageSearchIndex.add(entry.getMailboxSession(), entry.getMailbox(), message))
.thenReturn(Either.<Failure, Result>right(Result.COMPLETED))
.onErrorResume(e -> {
- LOGGER.warn("ReIndexing failed for {} {}", entry.getMailbox().generateAssociatedPath(), entry.getMessage().getUid(), e);
- return Mono.just(Either.left(new MessageFailure(entry.getMailbox().getMailboxId(), entry.getMessage().getUid())));
+ LOGGER.warn("ReIndexing failed for {} {}", entry.getMailbox().generateAssociatedPath(), entry.getUid(), e);
+ return Mono.just(Either.left(new MessageFailure(entry.getMailbox().getMailboxId(), entry.getUid())));
});
}
+ private Mono<Either<Failure, Result>> correctIfNeeded(ReIndexingEntry entry) {
+ MessageMapper messageMapper = mailboxSessionMapperFactory.getMessageMapper(entry.getMailboxSession());
+
+ return messageMapper.findInMailboxReactive(entry.getMailbox(), MessageRange.one(entry.getUid()), MessageMapper.FetchType.Metadata, ONE)
+ .next()
+ .flatMap(message -> isIndexUpToDate(entry.getMailbox(), message)
+ .flatMap(upToDate -> {
+ if (upToDate) {
+ return Mono.just(Either.right(Result.COMPLETED));
+ }
+ return correct(entry, message);
+ }));
+ }
+
+ private Mono<Either<Failure, Result>> correct(ReIndexingEntry entry, MailboxMessage message) {
+ return messageSearchIndex.delete(entry.getMailboxSession(), entry.getMailbox(), ImmutableList.of(message.getUid()))
+ .then(index(entry));
+ }
+
+ private Mono<Boolean> isIndexUpToDate(Mailbox mailbox, MailboxMessage message) {
+ return messageSearchIndex.retrieveIndexedFlags(mailbox, message.getUid())
+ .map(flags -> isIndexUpToDate(message, flags));
+ }
+
+ private boolean isIndexUpToDate(MailboxMessage message, Flags flags) {
+ return message.createFlags().equals(flags);
+ }
+
private <X, Y> Either<X, Y> flatten(Either<X, Either<X, Y>> nestedEither) {
return nestedEither.getOrElseGet(Either::left);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 07/10: JAMES-3255 Postmaster example values should
be valid
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 0ba8f1ef561c77ba0f62be693636ae4461dc659a
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jun 19 13:36:04 2020 +0700
JAMES-3255 Postmaster example values should be valid
---
.../run/guice/cassandra-ldap/destination/conf/mailetcontainer.xml | 4 +++-
.../cassandra-rabbitmq-ldap/destination/conf/mailetcontainer.xml | 4 +++-
.../run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml | 4 +++-
dockerfiles/run/guice/cassandra/destination/conf/mailetcontainer.xml | 4 +++-
dockerfiles/run/guice/jpa-smtp/destination/conf/mailetcontainer.xml | 4 +++-
dockerfiles/run/guice/jpa/destination/conf/mailetcontainer.xml | 4 +++-
dockerfiles/run/guice/memory/destination/conf/mailetcontainer.xml | 4 +++-
dockerfiles/run/spring/destination/conf/mailetcontainer.xml | 4 +++-
examples/custom-mailets/src/main/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
mpt/impl/smtp/cassandra/src/test/resources/mailetcontainer.xml | 2 +-
server/app/src/main/resources/mailetcontainer.xml | 4 +++-
.../container/cli-integration/src/test/resources/mailetcontainer.xml | 2 +-
.../guice/cassandra-guice/src/test/resources/mailetcontainer.xml | 2 +-
.../guice/cassandra-ldap-guice/src/test/resources/mailetcontainer.xml | 2 +-
.../cassandra-rabbitmq-guice/src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../container/guice/jpa-guice/src/test/resources/mailetcontainer.xml | 2 +-
.../guice/jpa-smtp-mariadb/src/test/resources/mailetcontainer.xml | 2 +-
.../container/guice/jpa-smtp/sample-configuration/mailetcontainer.xml | 4 +++-
.../container/guice/jpa-smtp/src/test/resources/mailetcontainer.xml | 2 +-
.../guice/memory-guice/sample-configuration/mailetcontainer.xml | 4 +++-
.../guice/memory-guice/src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
.../src/test/resources/mailetcontainer.xml | 2 +-
src/homepage/howTo/mail-processing.html | 2 +-
31 files changed, 53 insertions(+), 31 deletions(-)
diff --git a/dockerfiles/run/guice/cassandra-ldap/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/cassandra-ldap/destination/conf/mailetcontainer.xml
index 017d7c2..38a5ede 100644
--- a/dockerfiles/run/guice/cassandra-ldap/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/cassandra-ldap/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/mailetcontainer.xml
index 1e621ba..d66b290 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml
index 1e621ba..d66b290 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/cassandra/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/cassandra/destination/conf/mailetcontainer.xml
index 1e621ba..d66b290 100644
--- a/dockerfiles/run/guice/cassandra/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/cassandra/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/jpa-smtp/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/jpa-smtp/destination/conf/mailetcontainer.xml
index e0ee151..c7120f5 100644
--- a/dockerfiles/run/guice/jpa-smtp/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/jpa-smtp/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/jpa/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/jpa/destination/conf/mailetcontainer.xml
index 45f75ce..bab9e37 100644
--- a/dockerfiles/run/guice/jpa/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/jpa/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/guice/memory/destination/conf/mailetcontainer.xml b/dockerfiles/run/guice/memory/destination/conf/mailetcontainer.xml
index af849fd..716e625 100644
--- a/dockerfiles/run/guice/memory/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/guice/memory/destination/conf/mailetcontainer.xml
@@ -24,7 +24,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/dockerfiles/run/spring/destination/conf/mailetcontainer.xml b/dockerfiles/run/spring/destination/conf/mailetcontainer.xml
index 49237c8..4e07ba2 100644
--- a/dockerfiles/run/spring/destination/conf/mailetcontainer.xml
+++ b/dockerfiles/run/spring/destination/conf/mailetcontainer.xml
@@ -22,7 +22,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/examples/custom-mailets/src/main/resources/mailetcontainer.xml b/examples/custom-mailets/src/main/resources/mailetcontainer.xml
index d75ba9e..125b775 100644
--- a/examples/custom-mailets/src/main/resources/mailetcontainer.xml
+++ b/examples/custom-mailets/src/main/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster@localhost</postmaster>
</context>
<spooler>
diff --git a/mpt/impl/smtp/cassandra-rabbitmq-object-storage/src/test/resources/mailetcontainer.xml b/mpt/impl/smtp/cassandra-rabbitmq-object-storage/src/test/resources/mailetcontainer.xml
index 34d5c8b..7c58ed0 100644
--- a/mpt/impl/smtp/cassandra-rabbitmq-object-storage/src/test/resources/mailetcontainer.xml
+++ b/mpt/impl/smtp/cassandra-rabbitmq-object-storage/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@localhost</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/mpt/impl/smtp/cassandra/src/test/resources/mailetcontainer.xml b/mpt/impl/smtp/cassandra/src/test/resources/mailetcontainer.xml
index 34d5c8b..7c58ed0 100644
--- a/mpt/impl/smtp/cassandra/src/test/resources/mailetcontainer.xml
+++ b/mpt/impl/smtp/cassandra/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@localhost</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/app/src/main/resources/mailetcontainer.xml b/server/app/src/main/resources/mailetcontainer.xml
index e443e33..91dd386 100644
--- a/server/app/src/main/resources/mailetcontainer.xml
+++ b/server/app/src/main/resources/mailetcontainer.xml
@@ -31,7 +31,9 @@
<!-- MailAddress used for PostMaster -->
<context>
- <postmaster>postmaster@localhost</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster</postmaster>
</context>
<!-- Number of spool threads -->
diff --git a/server/container/cli-integration/src/test/resources/mailetcontainer.xml b/server/container/cli-integration/src/test/resources/mailetcontainer.xml
index 5322187..66307b0 100644
--- a/server/container/cli-integration/src/test/resources/mailetcontainer.xml
+++ b/server/container/cli-integration/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/cassandra-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/cassandra-guice/src/test/resources/mailetcontainer.xml
index 4549bd2..f22ddb7 100644
--- a/server/container/guice/cassandra-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/cassandra-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/cassandra-ldap-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/cassandra-ldap-guice/src/test/resources/mailetcontainer.xml
index c2f0cc0..2340ac6 100644
--- a/server/container/guice/cassandra-ldap-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/cassandra-ldap-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/cassandra-rabbitmq-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/cassandra-rabbitmq-guice/src/test/resources/mailetcontainer.xml
index 0f6b8dd..d629b8a 100644
--- a/server/container/guice/cassandra-rabbitmq-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/cassandra-rabbitmq-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/cassandra-rabbitmq-ldap-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/cassandra-rabbitmq-ldap-guice/src/test/resources/mailetcontainer.xml
index c2f0cc0..2340ac6 100644
--- a/server/container/guice/cassandra-rabbitmq-ldap-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/cassandra-rabbitmq-ldap-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/jpa-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/jpa-guice/src/test/resources/mailetcontainer.xml
index e03f3e5..0d6d961 100644
--- a/server/container/guice/jpa-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/jpa-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/jpa-smtp-mariadb/src/test/resources/mailetcontainer.xml b/server/container/guice/jpa-smtp-mariadb/src/test/resources/mailetcontainer.xml
index dedb391..5edda70 100644
--- a/server/container/guice/jpa-smtp-mariadb/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/jpa-smtp-mariadb/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@localhost</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/jpa-smtp/sample-configuration/mailetcontainer.xml b/server/container/guice/jpa-smtp/sample-configuration/mailetcontainer.xml
index 0317de4..0c05525 100644
--- a/server/container/guice/jpa-smtp/sample-configuration/mailetcontainer.xml
+++ b/server/container/guice/jpa-smtp/sample-configuration/mailetcontainer.xml
@@ -22,7 +22,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster@localhost</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/jpa-smtp/src/test/resources/mailetcontainer.xml b/server/container/guice/jpa-smtp/src/test/resources/mailetcontainer.xml
index dedb391..5edda70 100644
--- a/server/container/guice/jpa-smtp/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/jpa-smtp/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@localhost</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/memory-guice/sample-configuration/mailetcontainer.xml b/server/container/guice/memory-guice/sample-configuration/mailetcontainer.xml
index 9ed6c89..9d81841 100644
--- a/server/container/guice/memory-guice/sample-configuration/mailetcontainer.xml
+++ b/server/container/guice/memory-guice/sample-configuration/mailetcontainer.xml
@@ -22,7 +22,9 @@
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <!-- When the domain part of the postmaster mailAddress is missing, the default domain is appended.
+ You can configure it to (for example) <postmaster>postmaster@myDomain.com</postmaster> -->
+ <postmaster>postmaster@localhost</postmaster>
</context>
<spooler>
diff --git a/server/container/guice/memory-guice/src/test/resources/mailetcontainer.xml b/server/container/guice/memory-guice/src/test/resources/mailetcontainer.xml
index 7d68583..f14ec95 100644
--- a/server/container/guice/memory-guice/src/test/resources/mailetcontainer.xml
+++ b/server/container/guice/memory-guice/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/jmap-draft-integration-testing/cassandra-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml b/server/protocols/jmap-draft-integration-testing/cassandra-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
index f63f535..e6afe68 100644
--- a/server/protocols/jmap-draft-integration-testing/cassandra-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/jmap-draft-integration-testing/cassandra-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml b/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
index 849d564..47a6b82 100644
--- a/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/jmap-draft-integration-testing/rabbitmq-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml b/server/protocols/jmap-draft-integration-testing/rabbitmq-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
index f63f535..e6afe68 100644
--- a/server/protocols/jmap-draft-integration-testing/rabbitmq-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/jmap-draft-integration-testing/rabbitmq-jmap-draft-integration-testing/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/jmap-rfc-8621-integration-tests/distributed-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml b/server/protocols/jmap-rfc-8621-integration-tests/distributed-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
index f63f535..e6afe68 100644
--- a/server/protocols/jmap-rfc-8621-integration-tests/distributed-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/jmap-rfc-8621-integration-tests/distributed-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml b/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
index 849d564..47a6b82 100644
--- a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/resources/mailetcontainer.xml b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/resources/mailetcontainer.xml
index f8b5bfd..59b9b1e 100644
--- a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/resources/mailetcontainer.xml b/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/resources/mailetcontainer.xml
index 9c5c25d..e52366e 100644
--- a/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/resources/mailetcontainer.xml
+++ b/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/resources/mailetcontainer.xml
@@ -22,7 +22,7 @@
<mailetcontainer enableJmx="false">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster</postmaster>
</context>
<spooler>
diff --git a/src/homepage/howTo/mail-processing.html b/src/homepage/howTo/mail-processing.html
index ec41f2e..e1b82a3 100644
--- a/src/homepage/howTo/mail-processing.html
+++ b/src/homepage/howTo/mail-processing.html
@@ -173,7 +173,7 @@ public void init() throws MessagingException {
<mailetcontainer enableJmx="true">
<context>
- <postmaster>postmaster@james.minet.net</postmaster>
+ <postmaster>postmaster@localhost</postmaster>
</context>
<spooler>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 06/10: JAMES-3202 Update webadmin documentation
with the new Mode parameter in RunningOptions for reindexation tasks
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 54722ea65f3b03e3caf884735ab6bbdd23de3e2b
Author: Rene Cordier <rc...@linagora.com>
AuthorDate: Fri Jun 12 17:22:28 2020 +0700
JAMES-3202 Update webadmin documentation with the new Mode parameter in RunningOptions for reindexation tasks
---
src/site/markdown/server/manage-webadmin.md | 85 +++++++++++++++++++++++++----
1 file changed, 75 insertions(+), 10 deletions(-)
diff --git a/src/site/markdown/server/manage-webadmin.md b/src/site/markdown/server/manage-webadmin.md
index bf27154..7606c09 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -589,9 +589,21 @@ An admin can specify the concurrency that should be used when running the task:
This optional parameter must have a strictly positive integer as a value and be passed as query parameter.
+An admin can also specify the reindexing mode it wants to use when running the task:
+
+ - `mode` the reindexing mode used. There are 2 modes for the moment:
+ - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `fixOutdated` will check for outdated indexed document and reindex only those.
+
+This optional parameter must be passed as query parameter.
+
+It's good to note as well that there is a limitation with the `fixOutdated` mode. As we first collect metadata of
+stored messages to compare them with the ones in the index, a failed `expunged` operation might not be well corrected
+(as the message might not exist anymore but still be indexed).
+
Example:
-curl -XPOST http://ip:port/mailboxes?task=reIndex&messagesPerSecond=200
+curl -XPOST http://ip:port/mailboxes?task=reIndex&messagesPerSecond=200&mode=rebuildAll
The scheduled task will have the following type `full-reindexing` and the following `additionalInformation`:
@@ -599,7 +611,8 @@ The scheduled task will have the following type `full-reindexing` and the follow
{
"type":"full-reindexing",
"runningOptions":{
- "messagesPerSecond":200
+ "messagesPerSecond":200,
+ "mode":"REBUILD_ALL"
},
"successfullyReprocessedMailCount":18,
"failedReprocessedMailCount": 3,
@@ -630,10 +643,22 @@ An admin can specify the concurrency that should be used when running the task:
This optional parameter must have a strictly positive integer as a value and be passed as query parameter.
+An admin can also specify the reindexing mode it wants to use when running the task:
+
+ - `mode` the reindexing mode used. There are 2 modes for the moment:
+ - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `fixOutdated` will check for outdated indexed document and reindex only those.
+
+This optional parameter must be passed as query parameter.
+
+It's good to note as well that there is a limitation with the `fixOutdated` mode. As we first collect metadata of
+stored messages to compare them with the ones in the index, a failed `expunged` operation might not be well corrected
+(as the message might not exist anymore but still be indexed).
+
Example:
```
-curl -XPOST http://ip:port/mailboxes?task=reIndex&reIndexFailedMessagesOf=bbdb69c9-082a-44b0-a85a-6e33e74287a5&messagesPerSecond=200
+curl -XPOST http://ip:port/mailboxes?task=reIndex&reIndexFailedMessagesOf=bbdb69c9-082a-44b0-a85a-6e33e74287a5&messagesPerSecond=200&mode=rebuildAll
```
The scheduled task will have the following type `error-recovery-indexation` and the following `additionalInformation`:
@@ -642,7 +667,8 @@ The scheduled task will have the following type `error-recovery-indexation` and
{
"type":"error-recovery-indexation"
"runningOptions":{
- "messagesPerSecond":200
+ "messagesPerSecond":200,
+ "mode":"REBUILD_ALL"
},
"successfullyReprocessedMailCount":18,
"failedReprocessedMailCount": 3,
@@ -677,10 +703,22 @@ An admin can specify the concurrency that should be used when running the task:
This optional parameter must have a strictly positive integer as a value and be passed as query parameter.
+An admin can also specify the reindexing mode it wants to use when running the task:
+
+ - `mode` the reindexing mode used. There are 2 modes for the moment:
+ - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `fixOutdated` will check for outdated indexed document and reindex only those.
+
+This optional parameter must be passed as query parameter.
+
+It's good to note as well that there is a limitation with the `fixOutdated` mode. As we first collect metadata of
+stored messages to compare them with the ones in the index, a failed `expunged` operation might not be well corrected
+(as the message might not exist anymore but still be indexed).
+
Example:
```
-curl -XPOST http://ip:port/mailboxes/{mailboxId}?task=reIndex&messagesPerSecond=200
+curl -XPOST http://ip:port/mailboxes/{mailboxId}?task=reIndex&messagesPerSecond=200&mode=fixOutdated
```
Response codes:
@@ -694,7 +732,8 @@ The scheduled task will have the following type `mailbox-reindexing` and the fol
{
"type":"mailbox-reindexing",
"runningOptions":{
- "messagesPerSecond":200
+ "messagesPerSecond":200,
+ "mode":"FIX_OUTDATED"
},
"mailboxId":"{mailboxId}",
"successfullyReprocessedMailCount":18,
@@ -815,10 +854,22 @@ An admin can specify the concurrency that should be used when running the task:
This optional parameter must have a strictly positive integer as a value and be passed as query parameter.
+An admin can also specify the reindexing mode it wants to use when running the task:
+
+ - `mode` the reindexing mode used. There are 2 modes for the moment:
+ - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `fixOutdated` will check for outdated indexed document and reindex only those.
+
+This optional parameter must be passed as query parameter.
+
+It's good to note as well that there is a limitation with the `fixOutdated` mode. As we first collect metadata of
+stored messages to compare them with the ones in the index, a failed `expunged` operation might not be well corrected
+(as the message might not exist anymore but still be indexed).
+
Example:
```
-curl -XPOST /messages?task=SolveInconsistencies&messagesPerSecond=200
+curl -XPOST /messages?task=SolveInconsistencies&messagesPerSecond=200&mode=rebuildAll
```
Response codes:
@@ -838,7 +889,8 @@ The scheduled task will have the following type `solve-message-inconsistencies`
"updatedMessageIdEntries": 0,
"removedMessageIdEntries": 1,
"runningOptions":{
- "messagesPerSecond": 200
+ "messagesPerSecond": 200,
+ "mode":"REBUILD_ALL"
},
"fixedInconsistencies": [
{
@@ -1004,10 +1056,22 @@ An admin can specify the concurrency that should be used when running the task:
This optional parameter must have a strictly positive integer as a value and be passed as query parameter.
+An admin can also specify the reindexing mode it wants to use when running the task:
+
+ - `mode` the reindexing mode used. There are 2 modes for the moment:
+ - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `fixOutdated` will check for outdated indexed document and reindex only those.
+
+This optional parameter must be passed as query parameter.
+
+It's good to note as well that there is a limitation with the `fixOutdated` mode. As we first collect metadata of
+stored messages to compare them with the ones in the index, a failed `expunged` operation might not be well corrected
+(as the message might not exist anymore but still be indexed).
+
Example:
```
-curl -XPOST http://ip:port/users/{usernameToBeUsed}/mailboxes?task=reIndex&messagesPerSecond=200
+curl -XPOST http://ip:port/users/{usernameToBeUsed}/mailboxes?task=reIndex&messagesPerSecond=200&mode=fixOutdated
```
Response codes:
@@ -1021,7 +1085,8 @@ The scheduled task will have the following type `user-reindexing` and the follow
{
"type":"user-reindexing",
"runningOptions":{
- "messagesPerSecond":200
+ "messagesPerSecond":200,
+ "mode":"FIX_OUTDATED"
},
"user":"user@domain.com",
"successfullyReprocessedMailCount":18,
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 01/10: JAMES-3184 Transform throttle into a
reusable transformation
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit bd9147abd06475480f8fad41d3aba6d4d8169716
Author: Matthieu Baechler <ma...@apache.org>
AuthorDate: Fri May 29 11:30:28 2020 +0200
JAMES-3184 Transform throttle into a reusable transformation
---
.../task/SolveMessageInconsistenciesService.java | 24 +++++----
.../mailbox/tools/indexer/ReIndexerPerformer.java | 8 +--
.../quota/task/RecomputeCurrentQuotasService.java | 10 ++--
.../java/org/apache/james/util/ReactorUtils.java | 55 +++++++++----------
.../org/apache/james/util/ReactorUtilsTest.java | 62 +++++++++++++++-------
.../jmap/MessageFastViewProjectionCorrector.java | 7 +--
6 files changed, 97 insertions(+), 69 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
index 7d492fa..5279d56 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
@@ -423,11 +423,13 @@ public class SolveMessageInconsistenciesService {
}
private Flux<Task.Result> fixInconsistenciesInImapUid(Context context, RunningOptions runningOptions) {
- return ReactorUtils.Throttler.forOperation(this::detectInconsistencyInImapUid)
- .window(runningOptions.getMessagesPerSecond(), PERIOD)
- .throttle(messageIdToImapUidDAO.retrieveAllMessages())
- .doOnNext(any -> context.incrementProcessedImapUidEntries())
- .flatMap(inconsistency -> inconsistency.fix(context, messageIdToImapUidDAO, messageIdDAO));
+ return messageIdToImapUidDAO.retrieveAllMessages()
+ .transform(ReactorUtils.<ComposedMessageIdWithMetaData, Task.Result>throttle()
+ .elements(runningOptions.getMessagesPerSecond())
+ .per(PERIOD)
+ .forOperation(metaData -> detectInconsistencyInImapUid(metaData)
+ .doOnNext(any -> context.incrementProcessedImapUidEntries())
+ .flatMap(inconsistency -> inconsistency.fix(context, messageIdToImapUidDAO, messageIdDAO))));
}
private Mono<Inconsistency> detectInconsistencyInImapUid(ComposedMessageIdWithMetaData message) {
@@ -468,11 +470,13 @@ public class SolveMessageInconsistenciesService {
}
private Flux<Task.Result> fixInconsistenciesInMessageId(Context context, RunningOptions runningOptions) {
- return ReactorUtils.Throttler.forOperation(this::detectInconsistencyInMessageId)
- .window(runningOptions.getMessagesPerSecond(), PERIOD)
- .throttle(messageIdDAO.retrieveAllMessages())
- .doOnNext(any -> context.incrementMessageIdEntries())
- .flatMap(inconsistency -> inconsistency.fix(context, messageIdToImapUidDAO, messageIdDAO));
+ return messageIdDAO.retrieveAllMessages()
+ .transform(ReactorUtils.<ComposedMessageIdWithMetaData, Task.Result>throttle()
+ .elements(runningOptions.getMessagesPerSecond())
+ .per(PERIOD)
+ .forOperation(metadata -> detectInconsistencyInMessageId(metadata)
+ .doOnNext(any -> context.incrementMessageIdEntries())
+ .flatMap(inconsistency -> inconsistency.fix(context, messageIdToImapUidDAO, messageIdDAO))));
}
private Mono<Inconsistency> detectInconsistencyInMessageId(ComposedMessageIdWithMetaData message) {
diff --git a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
index 124238c..34d1a95 100644
--- a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
+++ b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
@@ -279,10 +279,10 @@ public class ReIndexerPerformer {
}
private Mono<Task.Result> reIndexMessages(Flux<Either<Failure, ReIndexingEntry>> entriesToIndex, RunningOptions runningOptions, ReprocessingContext reprocessingContext) {
- return ReactorUtils.Throttler.<Either<Failure, ReIndexingEntry>, Task.Result>forOperation(
- entry -> reIndex(entry, reprocessingContext))
- .window(runningOptions.getMessagesPerSecond(), Duration.ofSeconds(1))
- .throttle(entriesToIndex)
+ return entriesToIndex.transform(ReactorUtils.<Either<Failure, ReIndexingEntry>, Task.Result>throttle()
+ .elements(runningOptions.getMessagesPerSecond())
+ .per(Duration.ofSeconds(1))
+ .forOperation(entry -> reIndex(entry, reprocessingContext)))
.reduce(Task::combine)
.switchIfEmpty(Mono.just(Result.COMPLETED));
}
diff --git a/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
index 57d7b72..ef49c23 100644
--- a/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
+++ b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
@@ -47,7 +47,6 @@ import com.google.common.base.MoreObjects;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
-import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class RecomputeCurrentQuotasService {
@@ -164,10 +163,11 @@ public class RecomputeCurrentQuotasService {
public Mono<Task.Result> recomputeCurrentQuotas(Context context, RunningOptions runningOptions) {
try {
- Flux<Username> users = Iterators.toFlux(usersRepository.list());
- return ReactorUtils.Throttler.<Username, Task.Result>forOperation(username -> recomputeUserCurrentQuotas(context, username))
- .window(runningOptions.getUsersPerSecond(), Duration.ofSeconds(1))
- .throttle(users)
+ return Iterators.toFlux(usersRepository.list())
+ .transform(ReactorUtils.<Username, Task.Result>throttle()
+ .elements(runningOptions.getUsersPerSecond())
+ .per(Duration.ofSeconds(1))
+ .forOperation(username -> recomputeUserCurrentQuotas(context, username)))
.reduce(Task.Result.COMPLETED, Task::combine);
} catch (UsersRepositoryException e) {
LOGGER.error("Error while accessing users from repository", e);
diff --git a/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java b/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
index a7bed0e..1548d1d 100644
--- a/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
+++ b/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
@@ -44,38 +44,35 @@ public class ReactorUtils {
public static final String MDC_KEY_PREFIX = "MDC-";
- public static class Throttler<T, U> {
- private static final Duration DELAY = Duration.ZERO;
-
- public static <T, U> RequiresWindowingParameters<T, U> forOperation(Function<T, Publisher<U>> operation) {
- return (maxSize, duration) -> new Throttler<>(operation, maxSize, duration);
- }
-
- @FunctionalInterface
- public interface RequiresWindowingParameters<T, U> {
- Throttler<T, U> window(int maxSize, Duration duration);
- }
-
- private Throttler(Function<T, Publisher<U>> operation, int windowMaxSize, Duration windowDuration) {
- Preconditions.checkArgument(windowMaxSize > 0, "'windowMaxSize' must be strictly positive");
- Preconditions.checkArgument(!windowDuration.isNegative(), "'windowDuration' must be strictly positive");
- Preconditions.checkArgument(!windowDuration.isZero(), "'windowDuration' must be strictly positive");
+ private static final Duration DELAY = Duration.ZERO;
+
+ public static <T, U> RequiresQuantity<T, U> throttle() {
+ return elements -> duration -> operation -> {
+ Preconditions.checkArgument(elements > 0, "'windowMaxSize' must be strictly positive");
+ Preconditions.checkArgument(!duration.isNegative(), "'windowDuration' must be strictly positive");
+ Preconditions.checkArgument(!duration.isZero(), "'windowDuration' must be strictly positive");
+
+ return flux -> flux
+ .windowTimeout(elements, duration)
+ .zipWith(Flux.interval(DELAY, duration))
+ .flatMap(Tuple2::getT1, elements, elements)
+ .flatMap(operation, elements);
+ };
+ }
- this.operation = operation;
- this.windowMaxSize = windowMaxSize;
- this.windowDuration = windowDuration;
- }
+ @FunctionalInterface
+ public interface RequiresQuantity<T, U> {
+ RequiresPeriod<T, U> elements(int maxSize);
+ }
- private final Function<T, Publisher<U>> operation;
- private final int windowMaxSize;
- private final Duration windowDuration;
+ @FunctionalInterface
+ public interface RequiresPeriod<T, U> {
+ RequiresOperation<T, U> per(Duration duration);
+ }
- public Flux<U> throttle(Flux<T> flux) {
- return flux.windowTimeout(windowMaxSize, windowDuration)
- .zipWith(Flux.interval(DELAY, windowDuration))
- .flatMap(Tuple2::getT1)
- .flatMap(operation, windowMaxSize);
- }
+ @FunctionalInterface
+ public interface RequiresOperation<T, U> {
+ Function<Flux<T>, Flux<U>> forOperation(Function<T, Publisher<U>> operation);
}
public static <T> Mono<T> executeAndEmpty(Runnable runnable) {
diff --git a/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java b/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
index 984c165..dc286ad 100644
--- a/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
+++ b/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
@@ -19,6 +19,7 @@
package org.apache.james.util;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
@@ -34,7 +35,6 @@ import java.util.function.Function;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.RandomStringUtils;
-import org.apache.james.util.ReactorUtils.Throttler;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.reactivestreams.Publisher;
@@ -56,29 +56,37 @@ class ReactorUtilsTest {
class Throttling {
@Test
void windowShouldThrowWhenMaxSizeIsNegative() {
- assertThatThrownBy(() -> Throttler.forOperation(Mono::just)
- .window(-1, Duration.ofSeconds(1)))
+ assertThatThrownBy(() -> ReactorUtils.<Integer, Integer>throttle()
+ .elements(-1)
+ .per(Duration.ofSeconds(1))
+ .forOperation(Mono::just))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void windowShouldThrowWhenMaxSizeIsZero() {
- assertThatThrownBy(() -> Throttler.forOperation(Mono::just)
- .window(0, Duration.ofSeconds(1)))
+ assertThatThrownBy(() -> ReactorUtils.throttle()
+ .elements(0)
+ .per(Duration.ofSeconds(1))
+ .forOperation(Mono::just))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void windowShouldThrowWhenDurationIsNegative() {
- assertThatThrownBy(() -> Throttler.forOperation(Mono::just)
- .window(1, Duration.ofSeconds(-1)))
+ assertThatThrownBy(() -> ReactorUtils.throttle()
+ .elements(1)
+ .per(Duration.ofSeconds(-1))
+ .forOperation(Mono::just))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void windowShouldThrowWhenDurationIsZero() {
- assertThatThrownBy(() -> Throttler.forOperation(Mono::just)
- .window(1, Duration.ZERO))
+ assertThatThrownBy(() -> ReactorUtils.throttle()
+ .elements(1)
+ .per(Duration.ofSeconds(0))
+ .forOperation(Mono::just))
.isInstanceOf(IllegalArgumentException.class);
}
@@ -89,13 +97,13 @@ class ReactorUtilsTest {
Stopwatch stopwatch = Stopwatch.createUnstarted();
- Flux<Integer> originalFlux = Flux.range(0, 10);
- ImmutableList<Long> windowMembership = Throttler.<Integer, Long>forOperation(
- i -> Mono.fromCallable(() -> stopwatch.elapsed(TimeUnit.MILLISECONDS)))
- .window(windowMaxSize, windowDuration)
- .throttle(originalFlux)
- .doOnSubscribe(signal -> stopwatch.start())
+ ImmutableList<Long> windowMembership = Flux.range(0, 10)
+ .transform(ReactorUtils.<Integer, Long>throttle()
+ .elements(windowMaxSize)
+ .per(windowDuration)
+ .forOperation(i -> Mono.fromCallable(() -> stopwatch.elapsed(TimeUnit.MILLISECONDS))))
.map(i -> i / 100)
+ .doOnSubscribe(signal -> stopwatch.start())
.collect(Guavate.toImmutableList())
.block();
@@ -104,6 +112,22 @@ class ReactorUtilsTest {
}
@Test
+ void largeWindowShouldNotOverrunIntermediateBuffers() {
+ // windowMaxSize exceeds Queues.SMALL_BUFFER_SIZE & Queues.SMALL_BUFFER_SIZE (256 by default)
+ // Combined with slow operations, this ensures we are not filling up intermediate buffers.
+ int windowMaxSize = 3_000;
+ Duration windowDuration = Duration.ofMillis(100);
+
+ assertThatCode(() -> Flux.range(0, 10_000)
+ .transform(ReactorUtils.<Integer, Long>throttle()
+ .elements(windowMaxSize)
+ .per(windowDuration)
+ .forOperation(i -> Mono.delay(windowDuration.multipliedBy(2))))
+ .blockLast())
+ .doesNotThrowAnyException();
+ }
+
+ @Test
void throttleDownStreamConcurrencyShouldNotExceedWindowMaxSize() {
int windowMaxSize = 3;
Duration windowDuration = Duration.ofMillis(100);
@@ -116,9 +140,11 @@ class ReactorUtilsTest {
.flatMap(i -> Mono.delay(windowDuration.multipliedBy(2)).thenReturn(i))
.flatMap(i -> Mono.fromRunnable(ongoingProcessing::decrementAndGet).thenReturn(i));
- ImmutableList<Integer> ongoingProcessingUponComputationStart = Throttler.forOperation(longRunningOperation)
- .window(windowMaxSize, windowDuration)
- .throttle(originalFlux)
+ ImmutableList<Integer> ongoingProcessingUponComputationStart = originalFlux
+ .transform(ReactorUtils.<Integer, Integer>throttle()
+ .elements(windowMaxSize)
+ .per(windowDuration)
+ .forOperation(longRunningOperation))
.collect(Guavate.toImmutableList())
.block();
diff --git a/server/protocols/webadmin/webadmin-jmap/src/main/java/org/apache/james/webadmin/data/jmap/MessageFastViewProjectionCorrector.java b/server/protocols/webadmin/webadmin-jmap/src/main/java/org/apache/james/webadmin/data/jmap/MessageFastViewProjectionCorrector.java
index f23e13c..b54aaaf 100644
--- a/server/protocols/webadmin/webadmin-jmap/src/main/java/org/apache/james/webadmin/data/jmap/MessageFastViewProjectionCorrector.java
+++ b/server/protocols/webadmin/webadmin-jmap/src/main/java/org/apache/james/webadmin/data/jmap/MessageFastViewProjectionCorrector.java
@@ -215,9 +215,10 @@ public class MessageFastViewProjectionCorrector {
}
private Mono<Result> correctProjection(Flux<ProjectionEntry> entries, RunningOptions runningOptions, Progress progress) {
- return ReactorUtils.Throttler.<ProjectionEntry, Result>forOperation(entry -> correctProjection(entry, progress))
- .window(runningOptions.getMessagesPerSecond(), PERIOD)
- .throttle(entries)
+ return entries.transform(ReactorUtils.<ProjectionEntry, Task.Result>throttle()
+ .elements(runningOptions.getMessagesPerSecond())
+ .per(PERIOD)
+ .forOperation(entry -> correctProjection(entry, progress)))
.reduce(Task::combine)
.switchIfEmpty(Mono.just(Result.COMPLETED));
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 02/10: JAMES-3202
`ReindexingRunningOptionsParser.parse` is a static method,
can call it directly
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit d666723712dce91cf0d27f799963f8b6be872ff5
Author: Rene Cordier <rc...@linagora.com>
AuthorDate: Fri Jun 5 14:58:04 2020 +0700
JAMES-3202 `ReindexingRunningOptionsParser.parse` is a static method, can call it directly
---
.../src/main/java/org/apache/james/webadmin/routes/MailboxesRoutes.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MailboxesRoutes.java b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MailboxesRoutes.java
index 8fd0900..2f89499 100644
--- a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MailboxesRoutes.java
+++ b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MailboxesRoutes.java
@@ -109,7 +109,7 @@ public class MailboxesRoutes implements Routes {
boolean indexingCorrection = !Strings.isNullOrEmpty(request.queryParams(RE_INDEX_FAILED_MESSAGES_QUERY_PARAM));
if (indexingCorrection) {
IndexingDetailInformation indexingDetailInformation = retrieveIndexingExecutionDetails(previousReIndexingService, request);
- return reIndexer.reIndex(indexingDetailInformation.failures(), new ReindexingRunningOptionsParser().parse(request));
+ return reIndexer.reIndex(indexingDetailInformation.failures(), ReindexingRunningOptionsParser.parse(request));
}
return reIndexer.reIndex(ReindexingRunningOptionsParser.parse(request));
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 08/10: JAMES-3213 Change ICALToJsonAttribute
replyTo header source to Reply-To
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 6d2cf13fbaa116cdcc8bcce37c187297a037cd24
Author: Gautier DI FOLCO <gd...@linagora.com>
AuthorDate: Mon Jun 22 12:29:02 2020 +0200
JAMES-3213 Change ICALToJsonAttribute replyTo header source to Reply-To
---
.../java/org/apache/james/transport/mailets/ICALToJsonAttribute.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mailet/icalendar/src/main/java/org/apache/james/transport/mailets/ICALToJsonAttribute.java b/mailet/icalendar/src/main/java/org/apache/james/transport/mailets/ICALToJsonAttribute.java
index ed0ea90..07e013b 100644
--- a/mailet/icalendar/src/main/java/org/apache/james/transport/mailets/ICALToJsonAttribute.java
+++ b/mailet/icalendar/src/main/java/org/apache/james/transport/mailets/ICALToJsonAttribute.java
@@ -105,7 +105,7 @@ public class ICALToJsonAttribute extends GenericMailet {
public static final AttributeName DEFAULT_SOURCE = AttributeName.of(DEFAULT_SOURCE_ATTRIBUTE_NAME);
public static final AttributeName DEFAULT_RAW_SOURCE = AttributeName.of(DEFAULT_RAW_SOURCE_ATTRIBUTE_NAME);
public static final AttributeName DEFAULT_DESTINATION = AttributeName.of(DEFAULT_DESTINATION_ATTRIBUTE_NAME);
- public static final String REPLY_TO_HEADER_NAME = "replyTo";
+ public static final String REPLY_TO_HEADER_NAME = "Reply-To";
static {
ICal4JConfigurator.configure();
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org