You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by rc...@apache.org on 2020/10/14 02:31:26 UTC
[james-project] branch master updated (f41c3ac -> aeb633a)
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from f41c3ac JAMES-3406 About ElasticSearch consistency model.
new 1d5e459 JAMES-3406 Detail more multi-dc setup
new 9f6f55e Rename tests with conflicting names (tests exist with same name and package in server/mailet/mailets/src/test/java/org/apache/james/transport/mailets)
new 2573cc6 JAMES-3277 SetMessagesUpdateProcessor should read outbox once
new 8b81831 JAMES-3277 Avoid a double message read upon CassandraMessageIdMapper::setFlags
new 9264641 JAMES-3277 SetMessagesUpdateProcessor: read all metadata upfront
new 644e4e7 JAMES-3277 StoreMessageIdManager::setFlags should read mailboxes only once
new 49df4ab JAMES-3423 Specify different JWT public keys for JMAP and WebAdmin
new 96ce3ed JAMES-3424 Better document the use of JWT
new 33da147 JAMES-3409 CassandraMailboxPathV3DAO (with UidValidity)
new b4a0e8f JAMES-3409 CassandraMailboxMapper should rely on MailboxPath V3 DAO
new 8a3dad0 JAMES-3409 Migration for MailboxPath V3
new f1bbf77 JAMES-3409 Singleton binding for CassandraMailboxPathV3DAO
new 41a2b58 JAMES-3409 Fix warnings in MailboxMergingTaskRunner
new 44f50c2 JAMES-3409 Fix FixingGhostMailboxTest
new 7ca7ebc JAMES-3409 Fix solving mailboxes inconsistencies
new 7a87f19 JAMES-3409 FIX CassandraMailboxManagerConsistencyTest
new 0ae02fa JAMES-3409 FIX CassandraMailboxManagerConsistencyTest
new 4427954 JAMES-3409 FIX CassandraMailboxMapperTest
new 390f64f JAMES-3407 Read repair for mailbox entity
new 5bfa119 JAMES-3407 Configuration for read repairs
new a82a1cd JAMES-3407 Document mailbox read repair
new aeb633a JAMES-3407 Add mailbox.read.repair.chance as a comment in cassandra.properties examples
The 22 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG.md | 1 +
.../init/configuration/CassandraConfiguration.java | 32 +++-
.../versions/CassandraSchemaVersionManager.java | 2 +-
.../destination/conf/cassandra.properties | 1 +
.../destination/conf/webadmin.properties | 6 +
.../destination/conf/cassandra.properties | 1 +
.../destination/conf/webadmin.properties | 6 +
.../destination/conf/cassandra.properties | 1 +
.../destination/conf/webadmin.properties | 6 +
.../destination/conf/cassandra.properties | 1 +
.../cassandra/destination/conf/webadmin.properties | 6 +
.../memory/destination/conf/webadmin.properties | 6 +
.../architecture/consistency-model.adoc | 8 +-
.../pages/distributed/configure/cassandra.adoc | 4 +
.../servers/pages/distributed/configure/jmap.adoc | 22 +++
.../pages/distributed/configure/webadmin.adoc | 27 +++
.../org/apache/james/mailbox/MessageIdManager.java | 6 +
.../CassandraMailboxSessionMapperFactory.java | 7 +-
.../cassandra/mail/CassandraMailboxMapper.java | 154 +++++++++++----
...thV2DAO.java => CassandraMailboxPathV3DAO.java} | 56 +++---
.../cassandra/mail/CassandraMessageIdMapper.java | 31 ++-
...2Migration.java => MailboxPathV3Migration.java} | 34 ++--
...thV3MigrationTaskAdditionalInformationDTO.java} | 38 ++--
.../MailboxPathV3MigrationTaskDTO.java} | 37 ++--
.../cassandra/mail/task/ConflictingEntry.java | 5 +-
.../mail/task/MailboxMergingTaskRunner.java | 5 +-
.../task/SolveMailboxInconsistenciesService.java | 127 ++++++------
.../cassandra/modules/CassandraMailboxModule.java | 13 ++
...Table.java => CassandraMailboxPathV3Table.java} | 7 +-
.../CassandraMailboxManagerConsistencyTest.java | 16 +-
.../CassandraSubscriptionManagerTest.java | 3 +
.../cassandra/mail/CassandraMailboxMapperTest.java | 212 +++++++++++++++++++--
...est.java => CassandraMailboxPathV3DAOTest.java} | 62 +++---
.../mail/CassandraMessageIdMapperTest.java | 30 +++
.../mailbox/cassandra/mail/MailboxFixture.java | 6 +
.../mail/migration/MailboxPathV2MigrationTest.java | 54 +++++-
...ilboxPathV3MigrationTaskSerializationTest.java} | 23 ++-
...onTest.java => MailboxPathV3MigrationTest.java} | 79 ++------
.../SolveMailboxInconsistenciesServiceTest.java | 94 ++++-----
.../inmemory/mail/InMemoryMessageIdMapper.java | 16 ++
.../james/mailbox/store/StoreMessageIdManager.java | 44 ++++-
.../james/mailbox/store/mail/MessageIdMapper.java | 4 +
.../modules/mailbox/CassandraMailboxModule.java | 2 +
.../modules/webadmin/CassandraRoutesModule.java | 3 +
.../org/apache/james/MemoryJamesServerMain.java | 3 +-
.../org/apache/james/jmap/draft/JMAPModule.java | 9 +-
.../apache/james/modules/server/NoJwtModule.java | 15 +-
.../james/modules/server/WebAdminServerModule.java | 26 ++-
...nTest.java => SpamAssassinIntegrationTest.java} | 2 +-
...Test.java => ToRepositoryIngtegrationTest.java} | 2 +-
...> ToSenderDomainRepositoryIntegrationTest.java} | 2 +-
.../james/jmap/draft/methods/ReferenceUpdater.java | 9 +-
.../draft/methods/SetMessagesUpdateProcessor.java | 44 +++--
.../james/jmap/jwt/JWTAuthenticationStrategy.java | 5 +-
.../org/apache/james/jwt/JwtTokenVerifier.java | 18 +-
.../org/apache/james/jwt/PublicKeyProvider.java | 8 +-
.../rabbitmq/FixingGhostMailboxTest.java | 10 +-
.../rabbitmq/RabbitMQJwtFilterIntegrationTest.java | 8 +-
.../memory/MemoryJwtFilterIntegrationTest.java | 8 +-
.../james/webadmin/WebAdminConfiguration.java | 25 ++-
.../james/webadmin/authentication/JwtFilter.java | 5 +-
.../webadmin/authentication/JwtFilterTest.java | 2 +-
src/site/xdoc/server/config-cassandra.xml | 2 +
63 files changed, 1019 insertions(+), 482 deletions(-)
copy mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/{CassandraMailboxPathV2DAO.java => CassandraMailboxPathV3DAO.java} (83%)
copy mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/{MailboxPathV2Migration.java => MailboxPathV3Migration.java} (80%)
copy mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/{MailboxPathV2MigrationTaskAdditionalInformationDTO.java => MailboxPathV3MigrationTaskAdditionalInformationDTO.java} (71%)
copy mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/{task/SolveMailboxInconsistenciesTaskDTO.java => migration/MailboxPathV3MigrationTaskDTO.java} (57%)
copy mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/table/{CassandraMailboxPathV2Table.java => CassandraMailboxPathV3Table.java} (90%)
copy mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/{CassandraMailboxPathV2DAOTest.java => CassandraMailboxPathV3DAOTest.java} (67%)
copy mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/{MailboxPathV2MigrationTaskSerializationTest.java => MailboxPathV3MigrationTaskSerializationTest.java} (72%)
copy mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/{MailboxPathV2MigrationTest.java => MailboxPathV3MigrationTest.java} (59%)
rename server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/{SpamAssassinTest.java => SpamAssassinIntegrationTest.java} (99%)
rename server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/{ToRepositoryTest.java => ToRepositoryIngtegrationTest.java} (99%)
rename server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/{ToSenderDomainRepositoryTest.java => ToSenderDomainRepositoryIntegrationTest.java} (99%)
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 06/22: JAMES-3277 StoreMessageIdManager::setFlags
should read mailboxes only once
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 644e4e7839b556709ef8c8d5a96053cb0016abca
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Mon Oct 12 12:35:13 2020 +0700
JAMES-3277 StoreMessageIdManager::setFlags should read mailboxes only once
They were previously read 2 times:
- For right enforcement
- For retrieving information to dispatch
By reading the mailboxes upfront we can be reading them only once.
---
.../james/mailbox/store/StoreMessageIdManager.java | 27 +++++++++++++++++-----
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
index 1a29de9..74aebbb 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
@@ -119,12 +119,20 @@ public class StoreMessageIdManager implements MessageIdManager {
@Override
public void setFlags(Flags newState, MessageManager.FlagsUpdateMode replace, MessageId messageId, List<MailboxId> mailboxIds, MailboxSession mailboxSession) throws MailboxException {
MessageIdMapper messageIdMapper = mailboxSessionMapperFactory.getMessageIdMapper(mailboxSession);
+ MailboxMapper mailboxMapper = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession);
+
+ int concurrency = 4;
+ List<Mailbox> targetMailboxes = Flux.fromIterable(mailboxIds)
+ .flatMap(mailboxMapper::findMailboxById, concurrency)
+ .collect(Guavate.toImmutableList())
+ .subscribeOn(Schedulers.elastic())
+ .block();
- MailboxReactorUtils.block(assertRightsOnMailboxIds(mailboxIds, mailboxSession, Right.Write));
+ assertRightsOnMailboxes(targetMailboxes, mailboxSession, Right.Write);
Multimap<MailboxId, UpdatedFlags> updatedFlags = messageIdMapper.setFlags(messageId, mailboxIds, newState, replace);
for (Map.Entry<MailboxId, Collection<UpdatedFlags>> entry : updatedFlags.asMap().entrySet()) {
- dispatchFlagsChange(mailboxSession, entry.getKey(), ImmutableList.copyOf(entry.getValue()));
+ dispatchFlagsChange(mailboxSession, entry.getKey(), ImmutableList.copyOf(entry.getValue()), targetMailboxes);
}
}
@@ -352,16 +360,23 @@ public class StoreMessageIdManager implements MessageIdManager {
.then());
}
- private void dispatchFlagsChange(MailboxSession mailboxSession, MailboxId mailboxId, ImmutableList<UpdatedFlags> updatedFlags) throws MailboxException {
+ private void dispatchFlagsChange(MailboxSession mailboxSession, MailboxId mailboxId, ImmutableList<UpdatedFlags> updatedFlags,
+ List<Mailbox> knownMailboxes) throws MailboxException {
if (updatedFlags.stream().anyMatch(UpdatedFlags::flagsChanged)) {
- mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxById(mailboxId)
- .flatMap(mailbox -> eventBus.dispatch(EventFactory.flagsUpdated()
+ Mailbox mailbox = knownMailboxes.stream()
+ .filter(knownMailbox -> knownMailbox.getMailboxId().equals(mailboxId))
+ .findFirst()
+ .orElseGet(Throwing.supplier(() -> MailboxReactorUtils.block(mailboxSessionMapperFactory.getMailboxMapper(mailboxSession)
+ .findMailboxById(mailboxId)
+ .subscribeOn(Schedulers.elastic())))
+ .sneakyThrow());
+ eventBus.dispatch(EventFactory.flagsUpdated()
.randomEventId()
.mailboxSession(mailboxSession)
.mailbox(mailbox)
.updatedFlags(updatedFlags)
.build(),
- new MailboxIdRegistrationKey(mailboxId)))
+ new MailboxIdRegistrationKey(mailboxId))
.subscribeOn(Schedulers.elastic())
.block();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 05/22: JAMES-3277 SetMessagesUpdateProcessor: read
all metadata upfront
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 926464152a3d8098bce65c5474008dd692c24d31
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Mon Oct 12 12:25:19 2020 +0700
JAMES-3277 SetMessagesUpdateProcessor: read all metadata upfront
- Reading messagesV2 metadata level is not required thus can be skept
- messageMetadata mimics MessageManager recent API
- More importantly this allows grouping of the rights management, that is performed only once upon retrieving messages.
---
.../org/apache/james/mailbox/MessageIdManager.java | 6 +++++
.../cassandra/mail/CassandraMessageIdMapper.java | 6 +++++
.../mail/CassandraMessageIdMapperTest.java | 4 +++
.../inmemory/mail/InMemoryMessageIdMapper.java | 16 +++++++++++
.../james/mailbox/store/StoreMessageIdManager.java | 17 +++++++++---
.../james/mailbox/store/mail/MessageIdMapper.java | 4 +++
.../james/jmap/draft/methods/ReferenceUpdater.java | 9 +++----
.../draft/methods/SetMessagesUpdateProcessor.java | 31 +++++++++++++++-------
8 files changed, 74 insertions(+), 19 deletions(-)
diff --git a/mailbox/api/src/main/java/org/apache/james/mailbox/MessageIdManager.java b/mailbox/api/src/main/java/org/apache/james/mailbox/MessageIdManager.java
index 92895c4..26a53cb 100644
--- a/mailbox/api/src/main/java/org/apache/james/mailbox/MessageIdManager.java
+++ b/mailbox/api/src/main/java/org/apache/james/mailbox/MessageIdManager.java
@@ -27,6 +27,7 @@ import javax.mail.Flags;
import org.apache.james.mailbox.MessageManager.FlagsUpdateMode;
import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.DeleteResult;
import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.MailboxId;
@@ -39,6 +40,11 @@ import com.google.common.collect.ImmutableList;
import reactor.core.publisher.Flux;
public interface MessageIdManager {
+ default Publisher<ComposedMessageIdWithMetaData> messageMetadata(MessageId id, MailboxSession session) {
+ return messagesMetadata(ImmutableList.of(id), session);
+ }
+
+ Publisher<ComposedMessageIdWithMetaData> messagesMetadata(Collection<MessageId> id, MailboxSession session);
Set<MessageId> accessibleMessages(Collection<MessageId> messageIds, final MailboxSession mailboxSession) throws MailboxException;
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
index 771c276..7c09008 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
@@ -47,6 +47,7 @@ import org.apache.james.mailbox.store.mail.MessageMapper.FetchType;
import org.apache.james.mailbox.store.mail.ModSeqProvider;
import org.apache.james.mailbox.store.mail.model.MailboxMessage;
import org.apache.james.util.FunctionalUtils;
+import org.reactivestreams.Publisher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -110,6 +111,11 @@ public class CassandraMessageIdMapper implements MessageIdMapper {
.flatMap(this::keepMessageIfMailboxExists);
}
+ @Override
+ public Publisher<ComposedMessageIdWithMetaData> findMetadata(MessageId messageId) {
+ return imapUidDAO.retrieve((CassandraMessageId) messageId, Optional.empty());
+ }
+
private Flux<MailboxMessage> keepMessageIfMailboxExists(GroupedFlux<MailboxId, MailboxMessage> groupedFlux) {
CassandraId cassandraId = (CassandraId) groupedFlux.key();
return mailboxDAO.retrieveMailbox(cassandraId)
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
index fae47aa..fb69a32 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
@@ -25,12 +25,16 @@ import static org.assertj.core.api.Assertions.assertThat;
import java.util.List;
import java.util.Optional;
+import javax.mail.Flags;
+
import org.apache.james.backends.cassandra.CassandraCluster;
import org.apache.james.backends.cassandra.CassandraClusterExtension;
+import org.apache.james.backends.cassandra.StatementRecorder;
import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration;
import org.apache.james.core.Username;
import org.apache.james.mailbox.MailboxSession;
import org.apache.james.mailbox.MailboxSessionUtil;
+import org.apache.james.mailbox.MessageManager;
import org.apache.james.mailbox.cassandra.CassandraMailboxSessionMapperFactory;
import org.apache.james.mailbox.cassandra.TestCassandraMailboxSessionMapperFactory;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
diff --git a/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/mail/InMemoryMessageIdMapper.java b/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/mail/InMemoryMessageIdMapper.java
index c16968b..f01a0e2 100644
--- a/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/mail/InMemoryMessageIdMapper.java
+++ b/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/mail/InMemoryMessageIdMapper.java
@@ -30,6 +30,8 @@ import javax.mail.Flags;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.james.mailbox.MessageManager.FlagsUpdateMode;
import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
@@ -41,6 +43,7 @@ import org.apache.james.mailbox.store.mail.MailboxMapper;
import org.apache.james.mailbox.store.mail.MessageIdMapper;
import org.apache.james.mailbox.store.mail.MessageMapper;
import org.apache.james.mailbox.store.mail.model.MailboxMessage;
+import org.reactivestreams.Publisher;
import com.github.fge.lambdas.Throwing;
import com.github.steveash.guavate.Guavate;
@@ -66,6 +69,19 @@ public class InMemoryMessageIdMapper implements MessageIdMapper {
}
@Override
+ public Publisher<ComposedMessageIdWithMetaData> findMetadata(MessageId messageId) {
+ return mailboxMapper.list()
+ .flatMap(mailbox -> messageMapper.findInMailboxReactive(mailbox, MessageRange.all(), MessageMapper.FetchType.Full, UNLIMITED))
+ .map(message -> new ComposedMessageIdWithMetaData(
+ new ComposedMessageId(
+ message.getMailboxId(),
+ message.getMessageId(),
+ message.getUid()),
+ message.createFlags(),
+ message.getModSeq()));
+ }
+
+ @Override
public Flux<MailboxMessage> findReactive(Collection<MessageId> messageIds, MessageMapper.FetchType fetchType) {
return mailboxMapper.list()
.flatMap(mailbox -> messageMapper.findInMailboxReactive(mailbox, MessageRange.all(), fetchType, UNLIMITED))
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
index e84d747..1a29de9 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/StoreMessageIdManager.java
@@ -43,6 +43,7 @@ import org.apache.james.mailbox.events.MailboxIdRegistrationKey;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.MailboxNotFoundException;
import org.apache.james.mailbox.extension.PreDeletionHook;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.DeleteResult;
import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.Mailbox;
@@ -69,6 +70,7 @@ import org.apache.james.mailbox.store.mail.model.Message;
import org.apache.james.mailbox.store.mail.model.impl.SimpleMailboxMessage;
import org.apache.james.mailbox.store.quota.QuotaChecker;
import org.apache.james.util.FunctionalUtils;
+import org.reactivestreams.Publisher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -158,6 +160,17 @@ public class StoreMessageIdManager implements MessageIdManager {
.map(Throwing.function(messageResultConverter(fetchGroup)).sneakyThrow());
}
+ @Override
+ public Publisher<ComposedMessageIdWithMetaData> messagesMetadata(Collection<MessageId> ids, MailboxSession session) {
+ MessageIdMapper messageIdMapper = mailboxSessionMapperFactory.getMessageIdMapper(session);
+ int concurrency = 4;
+ return Flux.fromIterable(ids)
+ .flatMap(id -> Flux.from(messageIdMapper.findMetadata(id)), concurrency)
+ .groupBy(metaData -> metaData.getComposedMessageId().getMailboxId())
+ .filterWhen(groupedFlux -> hasRightsOnMailboxReactive(session, Right.Read).apply(groupedFlux.key()))
+ .flatMap(Function.identity());
+ }
+
private ImmutableSet<MailboxId> getAllowedMailboxIds(MailboxSession mailboxSession, List<MailboxMessage> messageList, Right... rights) throws MailboxException {
return MailboxReactorUtils.block(Flux.fromIterable(messageList)
.map(MailboxMessage::getMailboxId)
@@ -432,10 +445,6 @@ public class StoreMessageIdManager implements MessageIdManager {
return mailboxMessage -> mailboxIds.contains(mailboxMessage.getMailboxId());
}
- private Function<MailboxMessage, Mono<Boolean>> hasRightsOn(MailboxSession session, Right... rights) {
- return hasRightsOnMailboxReactive(session, rights).compose(MailboxMessage::getMailboxId);
- }
-
private Function<MailboxId, Mono<Boolean>> hasRightsOnMailboxReactive(MailboxSession session, Right... rights) {
return mailboxId -> Mono.from(rightManager.myRights(mailboxId, session))
.map(myRights -> myRights.contains(rights))
diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageIdMapper.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageIdMapper.java
index 6e422e2..5191435 100644
--- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageIdMapper.java
+++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/mail/MessageIdMapper.java
@@ -26,12 +26,14 @@ import javax.mail.Flags;
import org.apache.james.mailbox.MessageManager;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.MailboxNotFoundException;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
import org.apache.james.mailbox.model.UpdatedFlags;
import org.apache.james.mailbox.store.mail.MessageMapper.FetchType;
import org.apache.james.mailbox.store.mail.model.MailboxMessage;
+import org.reactivestreams.Publisher;
import com.google.common.collect.Multimap;
@@ -42,6 +44,8 @@ public interface MessageIdMapper {
List<MailboxMessage> find(Collection<MessageId> messageIds, FetchType fetchType);
+ Publisher<ComposedMessageIdWithMetaData> findMetadata(MessageId messageId);
+
default Flux<MailboxMessage> findReactive(Collection<MessageId> messageIds, FetchType fetchType) {
return Flux.fromIterable(find(messageIds, fetchType));
}
diff --git a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/ReferenceUpdater.java b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/ReferenceUpdater.java
index 2e48cd4..17c946a 100644
--- a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/ReferenceUpdater.java
+++ b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/ReferenceUpdater.java
@@ -32,12 +32,10 @@ import org.apache.james.mailbox.MailboxSession;
import org.apache.james.mailbox.MessageIdManager;
import org.apache.james.mailbox.MessageManager.FlagsUpdateMode;
import org.apache.james.mailbox.exception.MailboxException;
-import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.Header;
import org.apache.james.mailbox.model.Headers;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
-import org.apache.james.mailbox.model.MessageResult;
import org.apache.james.mailbox.model.MultimailboxesSearchQuery;
import org.apache.james.mailbox.model.SearchQuery;
import org.apache.james.util.streams.Iterators;
@@ -96,9 +94,10 @@ public class ReferenceUpdater {
.collectList().block();
try {
MessageId reference = Iterables.getOnlyElement(references);
- List<MailboxId> mailboxIds = messageIdManager.getMessage(reference, FetchGroup.MINIMAL, session).stream()
- .map(MessageResult::getMailboxId)
- .collect(Guavate.toImmutableList());
+ List<MailboxId> mailboxIds = Flux.from(messageIdManager.messageMetadata(reference, session))
+ .map(metaData -> metaData.getComposedMessageId().getMailboxId())
+ .collect(Guavate.toImmutableList())
+ .block();
messageIdManager.setFlags(flag, FlagsUpdateMode.ADD, reference, mailboxIds, session);
} catch (NoSuchElementException e) {
logger.info("Unable to find a message with this Mime Message Id: " + messageId);
diff --git a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
index 82d423e..2c74d40 100644
--- a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
+++ b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
@@ -24,6 +24,7 @@ import static org.apache.james.jmap.draft.methods.Method.JMAP_PREFIX;
import java.io.IOException;
import java.util.Collection;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
import java.util.Properties;
import java.util.Set;
@@ -54,6 +55,7 @@ import org.apache.james.mailbox.Role;
import org.apache.james.mailbox.SystemMailboxesProvider;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.OverQuotaException;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.mailbox.model.FetchGroup;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MailboxId.Factory;
@@ -72,6 +74,7 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Multimap;
import io.vavr.control.Try;
import reactor.core.publisher.Flux;
@@ -135,9 +138,15 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
}
private void prepareResponse(SetMessagesRequest request, MailboxSession mailboxSession, SetMessagesResponse.Builder responseBuilder, Set<MailboxId> outboxes) {
- request.buildUpdatePatches(updatePatchConverter).forEach((id, patch) -> {
+ Map<MessageId, UpdateMessagePatch> patches = request.buildUpdatePatches(updatePatchConverter);
+
+ Multimap<MessageId, ComposedMessageIdWithMetaData> messages = Flux.from(messageIdManager.messagesMetadata(patches.keySet(), mailboxSession))
+ .collect(Guavate.toImmutableListMultimap(metaData -> metaData.getComposedMessageId().getMessageId()))
+ .block();
+
+ patches.forEach((id, patch) -> {
if (patch.isValid()) {
- update(outboxes, id, patch, mailboxSession, responseBuilder);
+ update(outboxes, id, patch, mailboxSession, responseBuilder, messages);
} else {
handleInvalidRequest(responseBuilder, id, patch.getValidationErrors(), patch);
}
@@ -146,9 +155,11 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
}
private void update(Set<MailboxId> outboxes, MessageId messageId, UpdateMessagePatch updateMessagePatch, MailboxSession mailboxSession,
- SetMessagesResponse.Builder builder) {
+ SetMessagesResponse.Builder builder, Multimap<MessageId, ComposedMessageIdWithMetaData> metadata) {
try {
- List<MessageResult> messages = messageIdManager.getMessage(messageId, FetchGroup.MINIMAL, mailboxSession);
+ List<ComposedMessageIdWithMetaData> messages = Optional.ofNullable(metadata.get(messageId))
+ .map(ImmutableList::copyOf)
+ .orElse(ImmutableList.of());
assertValidUpdate(messages, updateMessagePatch, outboxes);
if (messages.isEmpty()) {
@@ -223,16 +234,16 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
}
}
- private void assertValidUpdate(List<MessageResult> messagesToBeUpdated,
+ private void assertValidUpdate(List<ComposedMessageIdWithMetaData> messagesToBeUpdated,
UpdateMessagePatch updateMessagePatch,
Set<MailboxId> outboxMailboxes) {
ImmutableList<MailboxId> previousMailboxes = messagesToBeUpdated.stream()
- .map(MessageResult::getMailboxId)
+ .map(metaData -> metaData.getComposedMessageId().getMailboxId())
.collect(Guavate.toImmutableList());
List<MailboxId> targetMailboxes = getTargetedMailboxes(previousMailboxes, updateMessagePatch);
boolean isDraft = messagesToBeUpdated.stream()
- .map(MessageResult::getFlags)
+ .map(ComposedMessageIdWithMetaData::getFlags)
.map(Keywords.lenientFactory()::fromFlags)
.reduce(new KeywordsCombiner())
.orElse(Keywords.DEFAULT_VALUE)
@@ -294,12 +305,12 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
.collect(Guavate.toImmutableSet());
}
- private Stream<MailboxException> updateFlags(MessageId messageId, UpdateMessagePatch updateMessagePatch, MailboxSession mailboxSession, MessageResult messageResult) {
+ private Stream<MailboxException> updateFlags(MessageId messageId, UpdateMessagePatch updateMessagePatch, MailboxSession mailboxSession, ComposedMessageIdWithMetaData message) {
try {
if (!updateMessagePatch.isFlagsIdentity()) {
messageIdManager.setFlags(
- updateMessagePatch.applyToState(messageResult.getFlags()),
- FlagsUpdateMode.REPLACE, messageId, ImmutableList.of(messageResult.getMailboxId()), mailboxSession);
+ updateMessagePatch.applyToState(message.getFlags()),
+ FlagsUpdateMode.REPLACE, messageId, ImmutableList.of(message.getComposedMessageId().getMailboxId()), mailboxSession);
}
return Stream.of();
} catch (MailboxException e) {
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 02/22: Rename tests with conflicting names (tests
exist with same name and package in
server/mailet/mailets/src/test/java/org/apache/james/transport/mailets)
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 9f6f55ead5e16715230a8b822604252c50f2f7dd
Author: Juhan Aasaru <Ju...@nortal.com>
AuthorDate: Mon Oct 12 13:58:52 2020 +0300
Rename tests with conflicting names (tests exist with same name and package in server/mailet/mailets/src/test/java/org/apache/james/transport/mailets)
---
.../mailets/{SpamAssassinTest.java => SpamAssassinIntegrationTest.java} | 2 +-
.../{ToRepositoryTest.java => ToRepositoryIngtegrationTest.java} | 2 +-
...RepositoryTest.java => ToSenderDomainRepositoryIntegrationTest.java} | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinTest.java b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinIntegrationTest.java
similarity index 99%
rename from server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinTest.java
rename to server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinIntegrationTest.java
index 179548d..833d598 100644
--- a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinTest.java
+++ b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/SpamAssassinIntegrationTest.java
@@ -55,7 +55,7 @@ import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
-public class SpamAssassinTest {
+public class SpamAssassinIntegrationTest {
private static final String SPAM_CONTENT = "XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X";
@ClassRule
diff --git a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryTest.java b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryIngtegrationTest.java
similarity index 99%
rename from server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryTest.java
rename to server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryIngtegrationTest.java
index 698a155..e2b7407 100644
--- a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryTest.java
+++ b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToRepositoryIngtegrationTest.java
@@ -51,7 +51,7 @@ import org.junit.rules.TemporaryFolder;
import io.restassured.specification.RequestSpecification;
-public class ToRepositoryTest {
+public class ToRepositoryIngtegrationTest {
private static final String RECIPIENT = "touser@" + DEFAULT_DOMAIN;
public static final MailRepositoryUrl CUSTOM_REPOSITORY = MailRepositoryUrl.from("memory://var/mail/custom/");
diff --git a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryTest.java b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryIntegrationTest.java
similarity index 99%
rename from server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryTest.java
rename to server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryIntegrationTest.java
index 76f6b62..a6d6432 100644
--- a/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryTest.java
+++ b/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ToSenderDomainRepositoryIntegrationTest.java
@@ -41,7 +41,7 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
-public class ToSenderDomainRepositoryTest {
+public class ToSenderDomainRepositoryIntegrationTest {
private static final String RECIPIENT = "touser@" + DEFAULT_DOMAIN;
private static final String CUSTOM_REPOSITORY_PREFIX = "memory://var/mail/custom/";
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 08/22: JAMES-3424 Better document the use of JWT
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 96ce3ed879081ab7c98943bf4097a28f80aa7e67
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Sun Oct 11 09:29:35 2020 +0700
JAMES-3424 Better document the use of JWT
- How to generate keys ?
- How to generate tokens?
- How to use these tokens to authentify requests
---
.../servers/pages/distributed/configure/jmap.adoc | 22 ++++++++++++++++++++++
.../pages/distributed/configure/webadmin.adoc | 22 ++++++++++++++++++++++
2 files changed, 44 insertions(+)
diff --git a/docs/modules/servers/pages/distributed/configure/jmap.adoc b/docs/modules/servers/pages/distributed/configure/jmap.adoc
index bc679ac..0b5cce7 100644
--- a/docs/modules/servers/pages/distributed/configure/jmap.adoc
+++ b/docs/modules/servers/pages/distributed/configure/jmap.adoc
@@ -36,3 +36,25 @@ This should not be the same keystore than the ones used by TLS based protocols.
Enabling *TRACE* on `org.apache.james.jmap.wire` enables reactor-netty wiretap, logging of
all incoming and outgoing requests, outgoing requests. This will log also potentially sensible information
like authentication credentials.
+
+== Generating a JWT key pair
+
+The Distributed server enforces the use of RSA-SHA-256.
+
+One can use OpenSSL to generate a JWT key pair :
+
+ # private key
+ openssl genrsa -out rs256-4096-private.rsa 4096
+ # public key
+ openssl rsa -in rs256-4096-private.rsa -pubout > rs256-4096-public.pem
+
+The private key can be used to generate JWT tokens, for instance
+using link:https://github.com/vandium-io/jwtgen[jwtgen]:
+
+ jwtgen -a RS256 -p rs256-4096-private.rsa 4096 -c "sub=bob@domain.tld" -e 3600 -V
+
+This token can then be passed as `Bearer` of the `Authorization` header :
+
+ curl -H "Authorization: Bearer $token" -XPOST http://127.0.0.1:80/jmap -d '...'
+
+The public key can be referenced as `jwt.publickeypem.url` of the `jmap.properties` configuration file.
diff --git a/docs/modules/servers/pages/distributed/configure/webadmin.adoc b/docs/modules/servers/pages/distributed/configure/webadmin.adoc
index 7b673e1..3e36c4f 100644
--- a/docs/modules/servers/pages/distributed/configure/webadmin.adoc
+++ b/docs/modules/servers/pages/distributed/configure/webadmin.adoc
@@ -64,3 +64,25 @@ needs to be on the classpath or in the ./extensions-jars folder. Read mode about
xref:distributed/extending/webadmin-routes.adoc[creating you own webadmin routes].
|===
+
+== Generating a JWT key pair
+
+The Distributed server enforces the use of RSA-SHA-256.
+
+One can use OpenSSL to generate a JWT key pair :
+
+ # private key
+ openssl genrsa -out rs256-4096-private.rsa 4096
+ # public key
+ openssl rsa -in rs256-4096-private.rsa -pubout > rs256-4096-public.pem
+
+The private key can be used to generate JWT tokens, for instance
+using link:https://github.com/vandium-io/jwtgen[jwtgen]:
+
+ jwtgen -a RS256 -p rs256-4096-private.rsa 4096 -c "sub=bob@domain.tld" -e 3600 -V
+
+This token can then be passed as `Bearer` of the `Authorization` header :
+
+ curl -H "Authorization: Bearer $token" -XGET http://127.0.0.1:8000/domains
+
+The public key can be referenced as `jwt.publickeypem.url` of the `jmap.properties` configuration file.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 20/22: JAMES-3407 Configuration for read repairs
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 5bfa119feda1dece37911650c1e62cd7843d1ae8
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Oct 8 16:46:22 2020 +0700
JAMES-3407 Configuration for read repairs
---
.../init/configuration/CassandraConfiguration.java | 32 ++++++++++++++++++++--
.../CassandraMailboxSessionMapperFactory.java | 2 +-
.../cassandra/mail/CassandraMailboxMapper.java | 9 ++++--
.../cassandra/mail/CassandraMailboxMapperTest.java | 3 +-
.../mail/migration/MailboxPathV2MigrationTest.java | 3 +-
5 files changed, 40 insertions(+), 9 deletions(-)
diff --git a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/configuration/CassandraConfiguration.java b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/configuration/CassandraConfiguration.java
index 42bbce0..1acf52d 100644
--- a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/configuration/CassandraConfiguration.java
+++ b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/configuration/CassandraConfiguration.java
@@ -39,6 +39,7 @@ public class CassandraConfiguration {
private static final org.slf4j.Logger LOGGER = LoggerFactory.getLogger(CassandraConfiguration.class);
public static final int DEFAULT_MESSAGE_CHUNK_SIZE_ON_READ = 100;
+ public static final float DEFAULT_MAILBOX_READ_REPAIR = 0.1f;
public static final int DEFAULT_EXPUNGE_BATCH_SIZE = 50;
public static final int DEFAULT_UPDATE_FLAGS_BATCH_SIZE = 20;
public static final int DEFAULT_FLAGS_UPDATE_MESSAGE_MAX_RETRY = 1000;
@@ -55,6 +56,7 @@ public class CassandraConfiguration {
public static final List<String> VALID_CONSISTENCY_LEVEL_REGULAR = ImmutableList.of("QUORUM", "LOCAL_QUORUM", "EACH_QUORUM");
public static final List<String> VALID_CONSISTENCY_LEVEL_LIGHTWEIGHT_TRANSACTION = ImmutableList.of("SERIAL", "LOCAL_SERIAL");
+ private static final String MAILBOX_READ_REPAIR = "mailbox.read.repair.chance";
private static final String MAILBOX_MAX_RETRY_ACL = "mailbox.max.retry.acl";
private static final String MAILBOX_MAX_RETRY_MODSEQ = "mailbox.max.retry.modseq";
private static final String MAILBOX_MAX_RETRY_UID = "mailbox.max.retry.uid";
@@ -85,6 +87,7 @@ public class CassandraConfiguration {
private Optional<Integer> messageAttachmentIdsReadTimeout = Optional.empty();
private Optional<String> consistencyLevelRegular = Optional.empty();
private Optional<String> consistencyLevelLightweightTransaction = Optional.empty();
+ private Optional<Float> mailboxReadRepair = Optional.empty();
public Builder messageReadChunkSize(int value) {
Preconditions.checkArgument(value > 0, "messageReadChunkSize needs to be strictly positive");
@@ -152,6 +155,13 @@ public class CassandraConfiguration {
return this;
}
+ public Builder mailboxReadRepair(float value) {
+ Preconditions.checkArgument(value >= 0, "mailboxReadRepair needs to be positive");
+ Preconditions.checkArgument(value <= 1, "mailboxReadRepair needs to be less or equal to 1");
+ this.mailboxReadRepair = Optional.of(value);
+ return this;
+ }
+
public Builder messageReadChunkSize(Optional<Integer> value) {
value.ifPresent(this::messageReadChunkSize);
return this;
@@ -207,6 +217,11 @@ public class CassandraConfiguration {
return this;
}
+ public Builder mailboxReadRepair(Optional<Float> value) {
+ value.ifPresent(this::mailboxReadRepair);
+ return this;
+ }
+
public Builder consistencyLevelRegular(String value) {
Preconditions.checkArgument(VALID_CONSISTENCY_LEVEL_REGULAR.contains(value),
"consistencyLevelRegular needs to be one of the following: " + String.join(", ", VALID_CONSISTENCY_LEVEL_REGULAR));
@@ -252,7 +267,8 @@ public class CassandraConfiguration {
attachmentV2MigrationReadTimeout.orElse(DEFAULT_ATTACHMENT_V2_MIGRATION_READ_TIMEOUT),
messageAttachmentIdsReadTimeout.orElse(DEFAULT_MESSAGE_ATTACHMENT_ID_MIGRATION_READ_TIMEOUT),
consistencyLevelRegular,
- consistencyLevelLightweightTransaction);
+ consistencyLevelLightweightTransaction,
+ mailboxReadRepair.orElse(DEFAULT_MAILBOX_READ_REPAIR));
}
}
@@ -288,6 +304,8 @@ public class CassandraConfiguration {
propertiesConfiguration.getString(CONSISTENCY_LEVEL_REGULAR)))
.consistencyLevelLightweightTransaction(Optional.ofNullable(
propertiesConfiguration.getString(CONSISTENCY_LEVEL_LIGHTWEIGHT_TRANSACTION)))
+ .mailboxReadRepair(Optional.ofNullable(
+ propertiesConfiguration.getFloat(MAILBOX_READ_REPAIR, null)))
.build();
}
@@ -304,13 +322,14 @@ public class CassandraConfiguration {
private final int messageAttachmentIdsReadTimeout;
private final String consistencyLevelRegular;
private final String consistencyLevelLightweightTransaction;
+ private final float mailboxReadRepair;
@VisibleForTesting
CassandraConfiguration(int aclMaxRetry, int messageReadChunkSize, int expungeChunkSize,
int flagsUpdateMessageIdMaxRetry, int flagsUpdateMessageMaxRetry,
int modSeqMaxRetry, int uidMaxRetry, int fetchNextPageInAdvanceRow,
int blobPartSize, final int attachmentV2MigrationReadTimeout, int messageAttachmentIdsReadTimeout,
- String consistencyLevelRegular, String consistencyLevelLightweightTransaction) {
+ String consistencyLevelRegular, String consistencyLevelLightweightTransaction, float mailboxReadRepair) {
this.aclMaxRetry = aclMaxRetry;
this.messageReadChunkSize = messageReadChunkSize;
this.expungeChunkSize = expungeChunkSize;
@@ -324,6 +343,11 @@ public class CassandraConfiguration {
this.messageAttachmentIdsReadTimeout = messageAttachmentIdsReadTimeout;
this.consistencyLevelRegular = consistencyLevelRegular;
this.consistencyLevelLightweightTransaction = consistencyLevelLightweightTransaction;
+ this.mailboxReadRepair = mailboxReadRepair;
+ }
+
+ public float getMailboxReadRepair() {
+ return mailboxReadRepair;
}
public int getBlobPartSize() {
@@ -390,6 +414,7 @@ public class CassandraConfiguration {
&& Objects.equals(this.flagsUpdateMessageMaxRetry, that.flagsUpdateMessageMaxRetry)
&& Objects.equals(this.modSeqMaxRetry, that.modSeqMaxRetry)
&& Objects.equals(this.uidMaxRetry, that.uidMaxRetry)
+ && Objects.equals(this.mailboxReadRepair, that.mailboxReadRepair)
&& Objects.equals(this.fetchNextPageInAdvanceRow, that.fetchNextPageInAdvanceRow)
&& Objects.equals(this.blobPartSize, that.blobPartSize)
&& Objects.equals(this.attachmentV2MigrationReadTimeout, that.attachmentV2MigrationReadTimeout)
@@ -405,7 +430,7 @@ public class CassandraConfiguration {
return Objects.hash(aclMaxRetry, messageReadChunkSize, expungeChunkSize, flagsUpdateMessageIdMaxRetry,
flagsUpdateMessageMaxRetry, modSeqMaxRetry, uidMaxRetry, fetchNextPageInAdvanceRow,
blobPartSize, attachmentV2MigrationReadTimeout, messageAttachmentIdsReadTimeout,
- consistencyLevelRegular, consistencyLevelLightweightTransaction);
+ consistencyLevelRegular, consistencyLevelLightweightTransaction, mailboxReadRepair);
}
@Override
@@ -418,6 +443,7 @@ public class CassandraConfiguration {
.add("flagsUpdateMessageMaxRetry", flagsUpdateMessageMaxRetry)
.add("modSeqMaxRetry", modSeqMaxRetry)
.add("fetchNextPageInAdvanceRow", fetchNextPageInAdvanceRow)
+ .add("mailboxReadRepair", mailboxReadRepair)
.add("uidMaxRetry", uidMaxRetry)
.add("blobPartSize", blobPartSize)
.add("attachmentV2MigrationReadTimeout", attachmentV2MigrationReadTimeout)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
index 9767519..3cd91fd 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
@@ -168,7 +168,7 @@ public class CassandraMailboxSessionMapperFactory extends MailboxSessionMapperFa
@Override
public MailboxMapper createMailboxMapper(MailboxSession mailboxSession) {
- return new CassandraMailboxMapper(mailboxDAO, mailboxPathDAO, mailboxPathV2DAO, mailboxPathV3DAO, userMailboxRightsDAO, aclMapper, versionManager);
+ return new CassandraMailboxMapper(mailboxDAO, mailboxPathDAO, mailboxPathV2DAO, mailboxPathV3DAO, userMailboxRightsDAO, aclMapper, versionManager, cassandraConfiguration);
}
@Override
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
index e6d868d..e918b8e 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
@@ -25,6 +25,7 @@ import java.time.Duration;
import javax.inject.Inject;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.james.backends.cassandra.init.configuration.CassandraConfiguration;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionManager;
import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.core.Username;
@@ -60,7 +61,6 @@ public class CassandraMailboxMapper implements MailboxMapper {
private static final Duration MIN_RETRY_BACKOFF = Duration.ofMillis(10);
private static final Duration MAX_RETRY_BACKOFF = Duration.ofMillis(1000);
private static final SchemaVersion MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION = new SchemaVersion(8);
- private static final float READ_REPAIR_CHANCE = 0.1f;
private final CassandraMailboxDAO mailboxDAO;
private final CassandraMailboxPathDAOImpl mailboxPathDAO;
@@ -69,6 +69,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
private final CassandraACLMapper cassandraACLMapper;
private final CassandraUserMailboxRightsDAO userMailboxRightsDAO;
private final CassandraSchemaVersionManager versionManager;
+ private final CassandraConfiguration cassandraConfiguration;
private final SecureRandom secureRandom;
@Inject
@@ -78,7 +79,8 @@ public class CassandraMailboxMapper implements MailboxMapper {
CassandraMailboxPathV3DAO mailboxPathV3DAO,
CassandraUserMailboxRightsDAO userMailboxRightsDAO,
CassandraACLMapper aclMapper,
- CassandraSchemaVersionManager versionManager) {
+ CassandraSchemaVersionManager versionManager,
+ CassandraConfiguration cassandraConfiguration) {
this.mailboxDAO = mailboxDAO;
this.mailboxPathDAO = mailboxPathDAO;
this.mailboxPathV2DAO = mailboxPathV2DAO;
@@ -86,6 +88,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
this.userMailboxRightsDAO = userMailboxRightsDAO;
this.cassandraACLMapper = aclMapper;
this.versionManager = versionManager;
+ this.cassandraConfiguration = cassandraConfiguration;
this.secureRandom = new SecureRandom();
}
@@ -134,7 +137,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
}
private boolean shouldReadRepair() {
- return secureRandom.nextFloat() < READ_REPAIR_CHANCE;
+ return secureRandom.nextFloat() < cassandraConfiguration.getMailboxReadRepair();
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
index 0be1024..2b9d1d7 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
@@ -119,7 +119,8 @@ class CassandraMailboxMapperTest {
mailboxPathV3DAO,
userMailboxRightsDAO,
aclMapper,
- new CassandraSchemaVersionManager(versionDAO));
+ new CassandraSchemaVersionManager(versionDAO),
+ CassandraConfiguration.DEFAULT_CONFIGURATION);
}
@Nested
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
index c0832ab..6a4ef9f 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
@@ -103,7 +103,8 @@ class MailboxPathV2MigrationTest {
userMailboxRightsDAO,
CassandraConfiguration.DEFAULT_CONFIGURATION,
cassandraCluster.getCassandraConsistenciesConfiguration()),
- new CassandraSchemaVersionManager(new CassandraSchemaVersionDAO(cassandra.getConf())));
+ new CassandraSchemaVersionManager(new CassandraSchemaVersionDAO(cassandra.getConf())),
+ CassandraConfiguration.DEFAULT_CONFIGURATION);
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 21/22: JAMES-3407 Document mailbox read repair
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit a82a1cd68eb20a0879f1a0030ccad12b2bb6e3a6
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Oct 8 16:50:08 2020 +0700
JAMES-3407 Document mailbox read repair
---
CHANGELOG.md | 1 +
docs/modules/servers/pages/distributed/configure/cassandra.adoc | 4 ++++
src/site/xdoc/server/config-cassandra.xml | 2 ++
3 files changed, 7 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 51f48e3..89b2e5e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
- JAMES-3266 Offer an option to disable ElasticSearch in Distributed James product
- JAMES-3202 Reindex only outdated documents with the Mode option set to CORRECT in reindexing tasks
- JAMES-3405 Expose metrics of Guice servers over HTTP - enables easy Prometheus metrics collection
+- JAMES-3407 Distributed server: Read-repairs for the mailbox entity
### Changed
- Switch to Java 11 for build and run
diff --git a/docs/modules/servers/pages/distributed/configure/cassandra.adoc b/docs/modules/servers/pages/distributed/configure/cassandra.adoc
index e97e177..d4bdafe 100644
--- a/docs/modules/servers/pages/distributed/configure/cassandra.adoc
+++ b/docs/modules/servers/pages/distributed/configure/cassandra.adoc
@@ -135,6 +135,10 @@ If you want more explanation about Cassandra configuration, you should visit the
|===
| Property name | explanation
+| mailbox.read.repair.chance
+| Optional. Defaults to 0.1 (10% chance).
+Must be between 0 and 1 (inclusive). Controls the probability of doing a read-repair upon mailbox read.
+
| mailbox.max.retry.acl
| Optional. Defaults to 1000.
Controls the number of retries upon Cassandra ACL updates.
diff --git a/src/site/xdoc/server/config-cassandra.xml b/src/site/xdoc/server/config-cassandra.xml
index e725404..23280c4 100644
--- a/src/site/xdoc/server/config-cassandra.xml
+++ b/src/site/xdoc/server/config-cassandra.xml
@@ -118,6 +118,8 @@
<dd>Optional. Defaults to 256.<br/>
If specified defines the Cassandra <a href="https://docs.datastax.com/en/developer/java-driver/3.5/manual/pooling/#acquisition-queue">maximum size of the connection pool queue</a>.</dd>
+ <dt><strong>mailbox.read.repair.chance</strong></dt>
+ <dd>Optional. Defaults to 0.1 (10% chance).<br/> Must be between 0 and 1 (inclusive). Controls the probability of doing a read-repair upon mailbox read.</dd>
<dt><strong>mailbox.max.retry.acl</strong></dt>
<dd>Optional. Defaults to 1000.<br/> Controls the number of retries upon Cassandra ACL updates.</dd>
<dt><strong>mailbox.max.retry.modseq</strong></dt>
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 07/22: JAMES-3423 Specify different JWT public keys
for JMAP and WebAdmin
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 49df4abb3246886e69a8ee486d18747c7d4fbda5
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Sun Oct 11 09:27:36 2020 +0700
JAMES-3423 Specify different JWT public keys for JMAP and WebAdmin
Default to previous behaviour (webadmin uses JMAP JWT key) when to key is specified
in webadmin configuration, in order to avoid breaking changes.
---
.../destination/conf/webadmin.properties | 6 +++++
.../destination/conf/webadmin.properties | 6 +++++
.../destination/conf/webadmin.properties | 6 +++++
.../cassandra/destination/conf/webadmin.properties | 6 +++++
.../memory/destination/conf/webadmin.properties | 6 +++++
.../pages/distributed/configure/webadmin.adoc | 5 +++++
.../org/apache/james/MemoryJamesServerMain.java | 3 ++-
.../org/apache/james/jmap/draft/JMAPModule.java | 9 ++++++--
.../apache/james/modules/server/NoJwtModule.java | 15 ++++++++-----
.../james/modules/server/WebAdminServerModule.java | 26 ++++++++++++++++++++--
.../james/jmap/jwt/JWTAuthenticationStrategy.java | 5 ++++-
.../org/apache/james/jwt/JwtTokenVerifier.java | 18 +++++++++------
.../org/apache/james/jwt/PublicKeyProvider.java | 8 +------
.../rabbitmq/RabbitMQJwtFilterIntegrationTest.java | 8 +++++--
.../memory/MemoryJwtFilterIntegrationTest.java | 8 +++++--
.../james/webadmin/WebAdminConfiguration.java | 25 ++++++++++++++++++---
.../james/webadmin/authentication/JwtFilter.java | 5 +++--
.../webadmin/authentication/JwtFilterTest.java | 2 +-
18 files changed, 131 insertions(+), 36 deletions(-)
diff --git a/dockerfiles/run/guice/cassandra-ldap/destination/conf/webadmin.properties b/dockerfiles/run/guice/cassandra-ldap/destination/conf/webadmin.properties
index ec014ea..ab7b2e4 100644
--- a/dockerfiles/run/guice/cassandra-ldap/destination/conf/webadmin.properties
+++ b/dockerfiles/run/guice/cassandra-ldap/destination/conf/webadmin.properties
@@ -37,6 +37,12 @@ https.enabled=false
# Defaults to false
#jwt.enabled=true
+#
+## If you wish to use OAuth authentication, you should provide a valid JWT public key.
+## The following entry specify the link to the URL of the public key file,
+## which should be a PEM format file.
+##
+#jwt.publickeypem.url=file://conf/jwt_publickey
# Defaults to false
#cors.enable=true
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/webadmin.properties b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/webadmin.properties
index ec014ea..ab7b2e4 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/webadmin.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/webadmin.properties
@@ -37,6 +37,12 @@ https.enabled=false
# Defaults to false
#jwt.enabled=true
+#
+## If you wish to use OAuth authentication, you should provide a valid JWT public key.
+## The following entry specify the link to the URL of the public key file,
+## which should be a PEM format file.
+##
+#jwt.publickeypem.url=file://conf/jwt_publickey
# Defaults to false
#cors.enable=true
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/webadmin.properties b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/webadmin.properties
index ec014ea..ab7b2e4 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/webadmin.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/webadmin.properties
@@ -37,6 +37,12 @@ https.enabled=false
# Defaults to false
#jwt.enabled=true
+#
+## If you wish to use OAuth authentication, you should provide a valid JWT public key.
+## The following entry specify the link to the URL of the public key file,
+## which should be a PEM format file.
+##
+#jwt.publickeypem.url=file://conf/jwt_publickey
# Defaults to false
#cors.enable=true
diff --git a/dockerfiles/run/guice/cassandra/destination/conf/webadmin.properties b/dockerfiles/run/guice/cassandra/destination/conf/webadmin.properties
index ec014ea..ab7b2e4 100644
--- a/dockerfiles/run/guice/cassandra/destination/conf/webadmin.properties
+++ b/dockerfiles/run/guice/cassandra/destination/conf/webadmin.properties
@@ -37,6 +37,12 @@ https.enabled=false
# Defaults to false
#jwt.enabled=true
+#
+## If you wish to use OAuth authentication, you should provide a valid JWT public key.
+## The following entry specify the link to the URL of the public key file,
+## which should be a PEM format file.
+##
+#jwt.publickeypem.url=file://conf/jwt_publickey
# Defaults to false
#cors.enable=true
diff --git a/dockerfiles/run/guice/memory/destination/conf/webadmin.properties b/dockerfiles/run/guice/memory/destination/conf/webadmin.properties
index 52cfffd..340315e 100644
--- a/dockerfiles/run/guice/memory/destination/conf/webadmin.properties
+++ b/dockerfiles/run/guice/memory/destination/conf/webadmin.properties
@@ -37,6 +37,12 @@ https.enabled=false
# Defaults to false
#jwt.enabled=true
+#
+## If you wish to use OAuth authentication, you should provide a valid JWT public key.
+## The following entry specify the link to the URL of the public key file,
+## which should be a PEM format file.
+##
+#jwt.publickeypem.url=file://conf/jwt_publickey
# Defaults to false
#cors.enable=true
diff --git a/docs/modules/servers/pages/distributed/configure/webadmin.adoc b/docs/modules/servers/pages/distributed/configure/webadmin.adoc
index 8e4b5c5..7b673e1 100644
--- a/docs/modules/servers/pages/distributed/configure/webadmin.adoc
+++ b/docs/modules/servers/pages/distributed/configure/webadmin.adoc
@@ -53,6 +53,11 @@ to get some examples and hints.
| https.trust.password
| Specify the truststore password (default: null)
+| jwt.publickeypem.url
+| Optional. JWT tokens allow request to bypass authentication. Path to the JWT public key.
+Defaults to the `jwt.publickeypem.url` value of `jmap.properties` file if unspecified
+(legacy behaviour)
+
| extensions.routes
| List of Routes specified as fully qualified class name that should be loaded in addition to your product routes list. Routes
needs to be on the classpath or in the ./extensions-jars folder. Read mode about
diff --git a/server/container/guice/memory-guice/src/main/java/org/apache/james/MemoryJamesServerMain.java b/server/container/guice/memory-guice/src/main/java/org/apache/james/MemoryJamesServerMain.java
index 32256c5..3fcbe09 100644
--- a/server/container/guice/memory-guice/src/main/java/org/apache/james/MemoryJamesServerMain.java
+++ b/server/container/guice/memory-guice/src/main/java/org/apache/james/MemoryJamesServerMain.java
@@ -48,6 +48,7 @@ import org.apache.james.modules.server.MailRepositoriesRoutesModule;
import org.apache.james.modules.server.MailboxRoutesModule;
import org.apache.james.modules.server.MailboxesExportRoutesModule;
import org.apache.james.modules.server.MemoryMailQueueModule;
+import org.apache.james.modules.server.NoJwtModule;
import org.apache.james.modules.server.RawPostDequeueDecoratorModule;
import org.apache.james.modules.server.SieveRoutesModule;
import org.apache.james.modules.server.SwaggerRoutesModule;
@@ -86,7 +87,7 @@ public class MemoryJamesServerMain implements JamesServerMain {
binder -> binder.bind(WebAdminConfiguration.class).toInstance(WebAdminConfiguration.TEST_CONFIGURATION));
public static final Module WEBADMIN_TESTING = Modules.override(WEBADMIN)
- .with(WEBADMIN_NO_AUTH_MODULE);
+ .with(WEBADMIN_NO_AUTH_MODULE, new NoJwtModule());
public static final Module PROTOCOLS = Modules.combine(
new IMAPServerModule(),
diff --git a/server/container/guice/protocols/jmap/src/main/java/org/apache/james/jmap/draft/JMAPModule.java b/server/container/guice/protocols/jmap/src/main/java/org/apache/james/jmap/draft/JMAPModule.java
index f31b10d..7acbe92 100644
--- a/server/container/guice/protocols/jmap/src/main/java/org/apache/james/jmap/draft/JMAPModule.java
+++ b/server/container/guice/protocols/jmap/src/main/java/org/apache/james/jmap/draft/JMAPModule.java
@@ -25,6 +25,8 @@ import java.util.EnumSet;
import java.util.Optional;
import java.util.stream.Stream;
+import javax.inject.Named;
+
import org.apache.commons.configuration2.Configuration;
import org.apache.commons.configuration2.ex.ConfigurationException;
import org.apache.commons.io.FileUtils;
@@ -40,6 +42,7 @@ import org.apache.james.jmap.mailet.VacationMailet;
import org.apache.james.jmap.mailet.filter.JMAPFiltering;
import org.apache.james.jmap.rfc8621.RFC8621MethodsModule;
import org.apache.james.jwt.JwtConfiguration;
+import org.apache.james.jwt.JwtTokenVerifier;
import org.apache.james.lifecycle.api.StartUpCheck;
import org.apache.james.mailbox.MailboxManager;
import org.apache.james.mailbox.MailboxManager.SearchCapabilities;
@@ -152,8 +155,10 @@ public class JMAPModule extends AbstractModule {
@Provides
@Singleton
- JwtConfiguration providesJwtConfiguration(JMAPDraftConfiguration jmapConfiguration) {
- return new JwtConfiguration(jmapConfiguration.getJwtPublicKeyPem());
+ @Named("jmap")
+ JwtTokenVerifier providesJwtTokenVerifier(JMAPDraftConfiguration jmapConfiguration) {
+ JwtConfiguration jwtConfiguration = new JwtConfiguration(jmapConfiguration.getJwtPublicKeyPem());
+ return JwtTokenVerifier.create(jwtConfiguration);
}
private Optional<String> loadPublicKey(FileSystem fileSystem, Optional<String> jwtPublickeyPemUrl) {
diff --git a/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/NoJwtModule.java b/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/NoJwtModule.java
index a500595..a64a2b2 100644
--- a/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/NoJwtModule.java
+++ b/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/NoJwtModule.java
@@ -19,16 +19,19 @@
package org.apache.james.modules.server;
-import java.util.Optional;
+import javax.inject.Named;
-import org.apache.james.jwt.JwtConfiguration;
+import org.apache.james.jwt.JwtTokenVerifier;
import com.google.inject.AbstractModule;
+import com.google.inject.Provides;
+import com.google.inject.Singleton;
public class NoJwtModule extends AbstractModule {
-
- @Override
- protected void configure() {
- bind(JwtConfiguration.class).toInstance(new JwtConfiguration(Optional.empty()));
+ @Provides
+ @Singleton
+ @Named("jmap")
+ JwtTokenVerifier failWhenFallbackToJMAPConfiguration() {
+ throw new RuntimeException("JMAP is not enabled thus we can not fallback to its JWT configuration");
}
}
diff --git a/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/WebAdminServerModule.java b/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/WebAdminServerModule.java
index a2625be..fb7a7d0 100644
--- a/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/WebAdminServerModule.java
+++ b/server/container/guice/protocols/webadmin/src/main/java/org/apache/james/modules/server/WebAdminServerModule.java
@@ -22,6 +22,7 @@ package org.apache.james.modules.server;
import static org.apache.james.webadmin.WebAdminConfiguration.DISABLED_CONFIGURATION;
import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Optional;
import java.util.Set;
@@ -29,6 +30,9 @@ import java.util.Set;
import javax.inject.Named;
import org.apache.commons.configuration2.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.james.filesystem.api.FileSystem;
+import org.apache.james.jwt.JwtConfiguration;
import org.apache.james.jwt.JwtTokenVerifier;
import org.apache.james.utils.ClassName;
import org.apache.james.utils.GuiceGenericLoader;
@@ -56,6 +60,7 @@ import com.github.fge.lambdas.Throwing;
import com.github.steveash.guavate.Guavate;
import com.google.common.collect.ImmutableList;
import com.google.inject.AbstractModule;
+import com.google.inject.Provider;
import com.google.inject.Provides;
import com.google.inject.Scopes;
import com.google.inject.Singleton;
@@ -108,7 +113,7 @@ public class WebAdminServerModule extends AbstractModule {
@Provides
@Singleton
- public WebAdminConfiguration provideWebAdminConfiguration(PropertiesProvider propertiesProvider) throws Exception {
+ public WebAdminConfiguration provideWebAdminConfiguration(FileSystem fileSystem, PropertiesProvider propertiesProvider) throws Exception {
try {
Configuration configurationFile = propertiesProvider.getConfiguration("webadmin");
@@ -124,6 +129,8 @@ public class WebAdminServerModule extends AbstractModule {
.urlCORSOrigin(configurationFile.getString("cors.origin", DEFAULT_NO_CORS_ORIGIN))
.host(configurationFile.getString("host", WebAdminConfiguration.DEFAULT_HOST))
.additionalRoutes(additionalRoutes)
+ .jwtPublicKeyPEM(loadPublicKey(fileSystem,
+ Optional.ofNullable(configurationFile.getString("jwt.publickeypem.url", null))))
.build();
} catch (FileNotFoundException e) {
LOGGER.info("No webadmin.properties file. Disabling WebAdmin interface.");
@@ -131,10 +138,14 @@ public class WebAdminServerModule extends AbstractModule {
}
}
+ private Optional<String> loadPublicKey(FileSystem fileSystem, Optional<String> jwtPublickeyPemUrl) {
+ return jwtPublickeyPemUrl.map(Throwing.function(url -> FileUtils.readFileToString(fileSystem.getFile(url), StandardCharsets.US_ASCII)));
+ }
+
@Provides
@Singleton
public AuthenticationFilter providesAuthenticationFilter(PropertiesProvider propertiesProvider,
- JwtTokenVerifier jwtTokenVerifier) throws Exception {
+ @Named("webadmin") JwtTokenVerifier.Factory jwtTokenVerifier) throws Exception {
try {
Configuration configurationFile = propertiesProvider.getConfiguration("webadmin");
if (configurationFile.getBoolean("jwt.enabled", DEFAULT_JWT_DISABLED)) {
@@ -146,6 +157,17 @@ public class WebAdminServerModule extends AbstractModule {
}
}
+ @Provides
+ @Singleton
+ @Named("webadmin")
+ JwtTokenVerifier.Factory providesJwtTokenVerifier(WebAdminConfiguration webAdminConfiguration,
+ @Named("jmap") Provider<JwtTokenVerifier> jmapTokenVerifier) {
+ return () -> webAdminConfiguration.getJwtPublicKey()
+ .map(keyPath -> new JwtConfiguration(Optional.of(keyPath)))
+ .map(JwtTokenVerifier::create)
+ .orElseGet(jmapTokenVerifier::get);
+ }
+
private Optional<TlsConfiguration> readHttpsConfiguration(Configuration configurationFile) {
boolean enabled = configurationFile.getBoolean("https.enabled", DEFAULT_HTTPS_DISABLED);
if (enabled) {
diff --git a/server/protocols/jmap/src/main/java/org/apache/james/jmap/jwt/JWTAuthenticationStrategy.java b/server/protocols/jmap/src/main/java/org/apache/james/jmap/jwt/JWTAuthenticationStrategy.java
index 5ffbec0..2dc0c8c 100644
--- a/server/protocols/jmap/src/main/java/org/apache/james/jmap/jwt/JWTAuthenticationStrategy.java
+++ b/server/protocols/jmap/src/main/java/org/apache/james/jmap/jwt/JWTAuthenticationStrategy.java
@@ -19,6 +19,7 @@
package org.apache.james.jmap.jwt;
import javax.inject.Inject;
+import javax.inject.Named;
import org.apache.james.core.Username;
import org.apache.james.jmap.exceptions.UnauthorizedException;
@@ -44,7 +45,9 @@ public class JWTAuthenticationStrategy implements AuthenticationStrategy {
@Inject
@VisibleForTesting
- public JWTAuthenticationStrategy(JwtTokenVerifier tokenManager, MailboxManager mailboxManager, UsersRepository usersRepository) {
+ public JWTAuthenticationStrategy(@Named("jmap") JwtTokenVerifier tokenManager,
+ MailboxManager mailboxManager,
+ UsersRepository usersRepository) {
this.tokenManager = tokenManager;
this.mailboxManager = mailboxManager;
this.usersRepository = usersRepository;
diff --git a/server/protocols/jwt/src/main/java/org/apache/james/jwt/JwtTokenVerifier.java b/server/protocols/jwt/src/main/java/org/apache/james/jwt/JwtTokenVerifier.java
index 1516295..f7f833c 100644
--- a/server/protocols/jwt/src/main/java/org/apache/james/jwt/JwtTokenVerifier.java
+++ b/server/protocols/jwt/src/main/java/org/apache/james/jwt/JwtTokenVerifier.java
@@ -18,12 +18,9 @@
****************************************************************/
package org.apache.james.jwt;
-import javax.inject.Inject;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import io.jsonwebtoken.Claims;
@@ -34,12 +31,19 @@ import io.jsonwebtoken.MalformedJwtException;
public class JwtTokenVerifier {
- private static Logger LOGGER = LoggerFactory.getLogger(JwtTokenVerifier.class);
+ public interface Factory {
+ JwtTokenVerifier create();
+ }
+
+ public static JwtTokenVerifier create(JwtConfiguration jwtConfiguration) {
+ PublicKeyProvider publicKeyProvider = new PublicKeyProvider(jwtConfiguration, new PublicKeyReader());
+ return new JwtTokenVerifier(publicKeyProvider);
+ }
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JwtTokenVerifier.class);
private final PublicKeyProvider pubKeyProvider;
- @Inject
- @VisibleForTesting
- JwtTokenVerifier(PublicKeyProvider pubKeyProvider) {
+ public JwtTokenVerifier(PublicKeyProvider pubKeyProvider) {
this.pubKeyProvider = pubKeyProvider;
}
diff --git a/server/protocols/jwt/src/main/java/org/apache/james/jwt/PublicKeyProvider.java b/server/protocols/jwt/src/main/java/org/apache/james/jwt/PublicKeyProvider.java
index 9ce7792..4f102eb 100644
--- a/server/protocols/jwt/src/main/java/org/apache/james/jwt/PublicKeyProvider.java
+++ b/server/protocols/jwt/src/main/java/org/apache/james/jwt/PublicKeyProvider.java
@@ -20,18 +20,12 @@ package org.apache.james.jwt;
import java.security.PublicKey;
-import javax.inject.Inject;
-
-import com.google.common.annotations.VisibleForTesting;
-
public class PublicKeyProvider {
private final JwtConfiguration jwtConfiguration;
private final PublicKeyReader reader;
- @Inject
- @VisibleForTesting
- PublicKeyProvider(JwtConfiguration jwtConfiguration, PublicKeyReader reader) {
+ public PublicKeyProvider(JwtConfiguration jwtConfiguration, PublicKeyReader reader) {
this.jwtConfiguration = jwtConfiguration;
this.reader = reader;
}
diff --git a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQJwtFilterIntegrationTest.java b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQJwtFilterIntegrationTest.java
index a96b4a3..e9c190d 100644
--- a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQJwtFilterIntegrationTest.java
+++ b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQJwtFilterIntegrationTest.java
@@ -26,7 +26,7 @@ import org.apache.james.DockerElasticSearchExtension;
import org.apache.james.JamesServerBuilder;
import org.apache.james.JamesServerExtension;
import org.apache.james.SearchConfiguration;
-import org.apache.james.jwt.JwtConfiguration;
+import org.apache.james.jwt.JwtTokenVerifier;
import org.apache.james.modules.AwsS3BlobStoreExtension;
import org.apache.james.modules.RabbitMQExtension;
import org.apache.james.modules.blobstore.BlobStoreConfiguration;
@@ -36,6 +36,8 @@ import org.apache.james.webadmin.integration.JwtFilterIntegrationTest;
import org.apache.james.webadmin.integration.WebadminIntegrationTestModule;
import org.junit.jupiter.api.extension.RegisterExtension;
+import com.google.inject.name.Names;
+
class RabbitMQJwtFilterIntegrationTest extends JwtFilterIntegrationTest {
@RegisterExtension
static JamesServerExtension testExtension = new JamesServerBuilder<CassandraRabbitMQJamesConfiguration>(tmpDir ->
@@ -54,7 +56,9 @@ class RabbitMQJwtFilterIntegrationTest extends JwtFilterIntegrationTest {
.extension(new RabbitMQExtension())
.server(configuration -> CassandraRabbitMQJamesServerMain.createServer(configuration)
.overrideWith(binder -> binder.bind(AuthenticationFilter.class).to(JwtFilter.class))
- .overrideWith(binder -> binder.bind(JwtConfiguration.class).toInstance(jwtConfiguration()))
+ .overrideWith(binder -> binder.bind(JwtTokenVerifier.Factory.class)
+ .annotatedWith(Names.named("webadmin"))
+ .toInstance(() -> JwtTokenVerifier.create(jwtConfiguration())))
.overrideWith(new WebadminIntegrationTestModule()))
.build();
}
diff --git a/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/memory/MemoryJwtFilterIntegrationTest.java b/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/memory/MemoryJwtFilterIntegrationTest.java
index c1aeca7..770b596 100644
--- a/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/memory/MemoryJwtFilterIntegrationTest.java
+++ b/server/protocols/webadmin-integration-test/memory-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/memory/MemoryJwtFilterIntegrationTest.java
@@ -22,13 +22,15 @@ package org.apache.james.webadmin.integration.memory;
import org.apache.james.JamesServerBuilder;
import org.apache.james.JamesServerExtension;
import org.apache.james.MemoryJamesServerMain;
-import org.apache.james.jwt.JwtConfiguration;
+import org.apache.james.jwt.JwtTokenVerifier;
import org.apache.james.webadmin.authentication.AuthenticationFilter;
import org.apache.james.webadmin.authentication.JwtFilter;
import org.apache.james.webadmin.integration.JwtFilterIntegrationTest;
import org.apache.james.webadmin.integration.WebadminIntegrationTestModule;
import org.junit.jupiter.api.extension.RegisterExtension;
+import com.google.inject.name.Names;
+
class MemoryJwtFilterIntegrationTest extends JwtFilterIntegrationTest {
@RegisterExtension
@@ -36,6 +38,8 @@ class MemoryJwtFilterIntegrationTest extends JwtFilterIntegrationTest {
.server(configuration -> MemoryJamesServerMain.createServer(configuration)
.overrideWith(new WebadminIntegrationTestModule())
.overrideWith(binder -> binder.bind(AuthenticationFilter.class).to(JwtFilter.class))
- .overrideWith(binder -> binder.bind(JwtConfiguration.class).toInstance(jwtConfiguration())))
+ .overrideWith(binder -> binder.bind(JwtTokenVerifier.Factory.class)
+ .annotatedWith(Names.named("webadmin"))
+ .toInstance(() -> JwtTokenVerifier.create(jwtConfiguration()))))
.build();
}
diff --git a/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/WebAdminConfiguration.java b/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/WebAdminConfiguration.java
index ecb5c1b..f5d2279 100644
--- a/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/WebAdminConfiguration.java
+++ b/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/WebAdminConfiguration.java
@@ -58,6 +58,17 @@ public class WebAdminConfiguration {
private Optional<String> urlCORSOrigin = Optional.empty();
private Optional<String> host = Optional.empty();
private ImmutableList.Builder<String> additionalRoutes = ImmutableList.builder();
+ private Optional<String> jwtPublicKey = Optional.empty();
+
+ public Builder jwtPublicKeyPEM(String jwtPublicKeyPEM) {
+ this.jwtPublicKey = Optional.of(jwtPublicKeyPEM);
+ return this;
+ }
+
+ public Builder jwtPublicKeyPEM(Optional<String> jwtPublicKeyPEM) {
+ jwtPublicKeyPEM.ifPresent(this::jwtPublicKeyPEM);
+ return this;
+ }
public Builder tls(TlsConfiguration tlsConfiguration) {
this.tlsConfiguration = Optional.of(tlsConfiguration);
@@ -130,7 +141,8 @@ public class WebAdminConfiguration {
enableCORS.orElse(DEFAULT_CORS_DISABLED),
urlCORSOrigin.orElse(CORS_ALL_ORIGINS),
host.orElse(DEFAULT_HOST),
- additionalRoutes.build());
+ additionalRoutes.build(),
+ jwtPublicKey);
}
}
@@ -141,10 +153,11 @@ public class WebAdminConfiguration {
private final String urlCORSOrigin;
private final String host;
private final List<String> additionalRoutes;
+ private final Optional<String> jwtPublicKey;
@VisibleForTesting
WebAdminConfiguration(boolean enabled, Optional<PortSupplier> port, Optional<TlsConfiguration> tlsConfiguration,
- boolean enableCORS, String urlCORSOrigin, String host, List<String> additionalRoutes) {
+ boolean enableCORS, String urlCORSOrigin, String host, List<String> additionalRoutes, Optional<String> jwtPublicKey) {
this.enabled = enabled;
this.port = port;
this.tlsConfiguration = tlsConfiguration;
@@ -152,6 +165,11 @@ public class WebAdminConfiguration {
this.urlCORSOrigin = urlCORSOrigin;
this.host = host;
this.additionalRoutes = additionalRoutes;
+ this.jwtPublicKey = jwtPublicKey;
+ }
+
+ public Optional<String> getJwtPublicKey() {
+ return jwtPublicKey;
}
public List<String> getAdditionalRoutes() {
@@ -195,6 +213,7 @@ public class WebAdminConfiguration {
&& Objects.equals(this.port, that.port)
&& Objects.equals(this.tlsConfiguration, that.tlsConfiguration)
&& Objects.equals(this.enableCORS, that.enableCORS)
+ && Objects.equals(this.jwtPublicKey, that.jwtPublicKey)
&& Objects.equals(this.urlCORSOrigin, that.urlCORSOrigin)
&& Objects.equals(this.host, that.host)
&& Objects.equals(this.additionalRoutes, that.additionalRoutes);
@@ -204,6 +223,6 @@ public class WebAdminConfiguration {
@Override
public final int hashCode() {
- return Objects.hash(enabled, port, tlsConfiguration, enableCORS, urlCORSOrigin, host, additionalRoutes);
+ return Objects.hash(enabled, port, tlsConfiguration, enableCORS, jwtPublicKey, urlCORSOrigin, host, additionalRoutes);
}
}
diff --git a/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/authentication/JwtFilter.java b/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/authentication/JwtFilter.java
index 28d8cd6..972569b 100644
--- a/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/authentication/JwtFilter.java
+++ b/server/protocols/webadmin/webadmin-core/src/main/java/org/apache/james/webadmin/authentication/JwtFilter.java
@@ -24,6 +24,7 @@ import static spark.Spark.halt;
import java.util.Optional;
import javax.inject.Inject;
+import javax.inject.Named;
import org.apache.james.jwt.JwtTokenVerifier;
import org.eclipse.jetty.http.HttpStatus;
@@ -39,8 +40,8 @@ public class JwtFilter implements AuthenticationFilter {
private final JwtTokenVerifier jwtTokenVerifier;
@Inject
- public JwtFilter(JwtTokenVerifier jwtTokenVerifier) {
- this.jwtTokenVerifier = jwtTokenVerifier;
+ public JwtFilter(@Named("webadmin") JwtTokenVerifier.Factory jwtTokenVerifierFactory) {
+ this.jwtTokenVerifier = jwtTokenVerifierFactory.create();
}
@Override
diff --git a/server/protocols/webadmin/webadmin-core/src/test/java/org/apache/james/webadmin/authentication/JwtFilterTest.java b/server/protocols/webadmin/webadmin-core/src/test/java/org/apache/james/webadmin/authentication/JwtFilterTest.java
index 05cc9da..d2d46a4 100644
--- a/server/protocols/webadmin/webadmin-core/src/test/java/org/apache/james/webadmin/authentication/JwtFilterTest.java
+++ b/server/protocols/webadmin/webadmin-core/src/test/java/org/apache/james/webadmin/authentication/JwtFilterTest.java
@@ -66,7 +66,7 @@ public class JwtFilterTest {
@Before
public void setUp() {
jwtTokenVerifier = mock(JwtTokenVerifier.class);
- jwtFilter = new JwtFilter(jwtTokenVerifier);
+ jwtFilter = new JwtFilter(() -> jwtTokenVerifier);
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 03/22: JAMES-3277 SetMessagesUpdateProcessor should
read outbox once
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 2573cc65495b2c43d06e10a59a53467f6ad704bb
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Mon Oct 12 06:30:00 2020 +0200
JAMES-3277 SetMessagesUpdateProcessor should read outbox once
A call remained, reading outbox for each updates entry
---
.../jmap/draft/methods/SetMessagesUpdateProcessor.java | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
index 5716a54..82d423e 100644
--- a/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
+++ b/server/protocols/jmap-draft/src/main/java/org/apache/james/jmap/draft/methods/SetMessagesUpdateProcessor.java
@@ -149,7 +149,7 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
SetMessagesResponse.Builder builder) {
try {
List<MessageResult> messages = messageIdManager.getMessage(messageId, FetchGroup.MINIMAL, mailboxSession);
- assertValidUpdate(messages, updateMessagePatch, mailboxSession);
+ assertValidUpdate(messages, updateMessagePatch, outboxes);
if (messages.isEmpty()) {
addMessageIdNotFoundToResponse(messageId, builder);
@@ -223,9 +223,9 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
}
}
- private void assertValidUpdate(List<MessageResult> messagesToBeUpdated, UpdateMessagePatch updateMessagePatch, MailboxSession session) throws MailboxException {
- List<MailboxId> outboxMailboxes = mailboxIdFor(Role.OUTBOX, session);
-
+ private void assertValidUpdate(List<MessageResult> messagesToBeUpdated,
+ UpdateMessagePatch updateMessagePatch,
+ Set<MailboxId> outboxMailboxes) {
ImmutableList<MailboxId> previousMailboxes = messagesToBeUpdated.stream()
.map(MessageResult::getMailboxId)
.collect(Guavate.toImmutableList());
@@ -268,13 +268,6 @@ public class SetMessagesUpdateProcessor implements SetMessagesProcessor {
.orElse(previousMailboxes);
}
- private List<MailboxId> mailboxIdFor(Role role, MailboxSession session) throws MailboxException {
- return Flux.from(systemMailboxesProvider.getMailboxByRole(role, session.getUser()))
- .toStream()
- .map(MessageManager::getId)
- .collect(Guavate.toImmutableList());
- }
-
private MailImpl buildMailFromMessage(MessageResult message) throws MessagingException, IOException, MailboxException {
return MailImpl.fromMimeMessage(message.getMessageId().serialize(),
new MimeMessage(
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 16/22: JAMES-3409 FIX
CassandraMailboxManagerConsistencyTest
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7a87f19bd59fd0f9e5ca871b3b8d4be4183d7b29
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue Aug 11 17:03:05 2020 +0700
JAMES-3409 FIX CassandraMailboxManagerConsistencyTest
---
.../mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
index a81c8c6..ec299f9 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
@@ -424,7 +424,7 @@ class CassandraMailboxManagerConsistencyTest {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3"));
doQuietly(() -> testee.deleteMailbox(inboxPath, mailboxSession));
@@ -448,7 +448,7 @@ class CassandraMailboxManagerConsistencyTest {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3"));
doQuietly(() -> testee.deleteMailbox(inboxId, mailboxSession));
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 15/22: JAMES-3409 Fix solving mailboxes
inconsistencies
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7ca7ebcfd9e1d0c0596be8172aa226ac9ea8b1c2
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue Aug 11 15:43:27 2020 +0700
JAMES-3409 Fix solving mailboxes inconsistencies
---
.../cassandra/mail/task/ConflictingEntry.java | 5 +-
.../task/SolveMailboxInconsistenciesService.java | 94 +++++++++++-----------
.../SolveMailboxInconsistenciesServiceTest.java | 94 +++++++++++-----------
3 files changed, 95 insertions(+), 98 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/ConflictingEntry.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/ConflictingEntry.java
index 34a9f1a..87074c2 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/ConflictingEntry.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/ConflictingEntry.java
@@ -21,7 +21,6 @@ package org.apache.james.mailbox.cassandra.mail.task;
import java.util.Objects;
-import org.apache.james.mailbox.cassandra.mail.CassandraIdAndPath;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MailboxPath;
@@ -96,8 +95,8 @@ public class ConflictingEntry {
interface RequireMailboxPathDaoEntry {
ConflictingEntry mailboxPathDaoEntry(DaoEntry daoEntry);
- default ConflictingEntry mailboxPathDaoEntry(CassandraIdAndPath mailbox) {
- return mailboxPathDaoEntry(mailbox.getMailboxPath(), mailbox.getCassandraId());
+ default ConflictingEntry mailboxPathDaoEntry(Mailbox mailbox) {
+ return mailboxPathDaoEntry(mailbox.generateAssociatedPath(), mailbox.getMailboxId());
}
default ConflictingEntry mailboxPathDaoEntry(MailboxPath path, MailboxId id) {
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
index aa53506..8aacb18 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
@@ -30,9 +30,8 @@ import javax.inject.Inject;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionManager;
import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
-import org.apache.james.mailbox.cassandra.mail.CassandraIdAndPath;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
-import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.task.Task;
@@ -52,7 +51,7 @@ public class SolveMailboxInconsistenciesService {
@FunctionalInterface
interface Inconsistency {
- Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV2DAO pathV2DAO);
+ Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO pathV3DAO);
}
private static Inconsistency NO_INCONSISTENCY = (context, mailboxDAO1, pathV2DAO) -> Mono.just(Result.COMPLETED);
@@ -71,8 +70,8 @@ public class SolveMailboxInconsistenciesService {
}
@Override
- public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV2DAO pathV2DAO) {
- return pathV2DAO.save(mailbox.generateAssociatedPath(), (CassandraId) mailbox.getMailboxId())
+ public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO pathV3DAO) {
+ return pathV3DAO.save(mailbox)
.map(success -> {
if (success) {
notifySuccess(context);
@@ -112,27 +111,27 @@ public class SolveMailboxInconsistenciesService {
* In order to solve this inconsistency, we can simply re-reference the mailboxPath.
*/
private static class OrphanMailboxPathDAOEntry implements Inconsistency {
- private final CassandraIdAndPath pathRegistration;
+ private final Mailbox mailbox;
- private OrphanMailboxPathDAOEntry(CassandraIdAndPath pathRegistration) {
- this.pathRegistration = pathRegistration;
+ private OrphanMailboxPathDAOEntry(Mailbox mailbox) {
+ this.mailbox = mailbox;
}
@Override
- public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV2DAO pathV2DAO) {
- return pathV2DAO.delete(pathRegistration.getMailboxPath())
+ public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO pathV3DAO) {
+ return pathV3DAO.delete(mailbox.generateAssociatedPath())
.doOnSuccess(any -> {
LOGGER.info("Inconsistency fixed for orphan mailboxPath {} - {}",
- pathRegistration.getCassandraId().serialize(),
- pathRegistration.getMailboxPath().asString());
- context.addFixedInconsistency(pathRegistration.getCassandraId());
+ mailbox.getMailboxId().serialize(),
+ mailbox.generateAssociatedPath().asString());
+ context.addFixedInconsistency(mailbox.getMailboxId());
})
.map(any -> Result.COMPLETED)
.defaultIfEmpty(Result.COMPLETED)
.onErrorResume(e -> {
LOGGER.error("Failed fixing inconsistency for orphan mailboxPath {} - {}",
- pathRegistration.getCassandraId().serialize(),
- pathRegistration.getMailboxPath().asString(),
+ mailbox.getMailboxId().serialize(),
+ mailbox.generateAssociatedPath().asString(),
e);
context.incrementErrors();
return Mono.just(Result.PARTIAL);
@@ -152,19 +151,12 @@ public class SolveMailboxInconsistenciesService {
private static class ConflictingEntryInconsistency implements Inconsistency {
private final ConflictingEntry conflictingEntry;
- private ConflictingEntryInconsistency(Mailbox mailbox, CassandraIdAndPath pathRegistration) {
- boolean samePath = mailbox.generateAssociatedPath().equals(pathRegistration.getMailboxPath());
- boolean sameId = mailbox.getMailboxId().equals(pathRegistration.getCassandraId());
-
- Preconditions.checkState(samePath != sameId);
-
- this.conflictingEntry = ConflictingEntry.builder()
- .mailboxDaoEntry(mailbox)
- .mailboxPathDaoEntry(pathRegistration);
+ private ConflictingEntryInconsistency(ConflictingEntry conflictingEntry) {
+ this.conflictingEntry = conflictingEntry;
}
@Override
- public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV2DAO pathV2DAO) {
+ public Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO pathV3DAO) {
LOGGER.error("MailboxDAO contains mailbox {} {} which conflict with corresponding registration {} {}. " +
"We recommend merging these mailboxes together to prevent mail data loss.",
conflictingEntry.getMailboxDaoEntry().getMailboxId(), conflictingEntry.getMailboxDaoEntry().getMailboxPath(),
@@ -353,16 +345,16 @@ public class SolveMailboxInconsistenciesService {
}
}
- private static final SchemaVersion MAILBOX_PATH_V_2_MIGRATION_PERFORMED_VERSION = new SchemaVersion(6);
+ private static final SchemaVersion MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION = new SchemaVersion(8);
private final CassandraMailboxDAO mailboxDAO;
- private final CassandraMailboxPathV2DAO mailboxPathV2DAO;
+ private final CassandraMailboxPathV3DAO mailboxPathV3DAO;
private final CassandraSchemaVersionManager versionManager;
@Inject
- SolveMailboxInconsistenciesService(CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV2DAO mailboxPathV2DAO, CassandraSchemaVersionManager versionManager) {
+ SolveMailboxInconsistenciesService(CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO mailboxPathV3DAO, CassandraSchemaVersionManager versionManager) {
this.mailboxDAO = mailboxDAO;
- this.mailboxPathV2DAO = mailboxPathV2DAO;
+ this.mailboxPathV3DAO = mailboxPathV3DAO;
this.versionManager = versionManager;
}
@@ -377,49 +369,55 @@ public class SolveMailboxInconsistenciesService {
private void assertValidVersion() {
SchemaVersion version = versionManager.computeVersion().block();
- boolean isVersionValid = version.isAfterOrEquals(MAILBOX_PATH_V_2_MIGRATION_PERFORMED_VERSION);
+ boolean isVersionValid = version.isAfterOrEquals(MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION);
Preconditions.checkState(isVersionValid,
- "Schema version %s is required in order to ensure mailboxPathV2DAO to be correctly populated, got %s",
- MAILBOX_PATH_V_2_MIGRATION_PERFORMED_VERSION.getValue(),
+ "Schema version %s is required in order to ensure mailboxPathV3DAO to be correctly populated, got %s",
+ MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION.getValue(),
version.getValue());
}
private Flux<Result> processMailboxPathDaoInconsistencies(Context context) {
- return mailboxPathV2DAO.listAll()
- .flatMap(this::detectInconsistency)
- .flatMap(inconsistency -> inconsistency.fix(context, mailboxDAO, mailboxPathV2DAO))
+ return mailboxPathV3DAO.listAll()
+ .flatMap(this::detectMailboxPathDaoInconsistency)
+ .flatMap(inconsistency -> inconsistency.fix(context, mailboxDAO, mailboxPathV3DAO))
.doOnNext(any -> context.incrementProcessedMailboxPathEntries());
}
private Flux<Result> processMailboxDaoInconsistencies(Context context) {
return mailboxDAO.retrieveAllMailboxes()
- .flatMap(this::detectInconsistency)
- .flatMap(inconsistency -> inconsistency.fix(context, mailboxDAO, mailboxPathV2DAO))
+ .flatMap(this::detectMailboxDaoInconsistency)
+ .flatMap(inconsistency -> inconsistency.fix(context, mailboxDAO, mailboxPathV3DAO))
.doOnNext(any -> context.incrementProcessedMailboxEntries());
}
- private Mono<Inconsistency> detectInconsistency(Mailbox mailbox) {
- return mailboxPathV2DAO.retrieveId(mailbox.generateAssociatedPath())
- .map(pathRegistration -> {
- if (pathRegistration.getCassandraId().equals(mailbox.getMailboxId())) {
+ private Mono<Inconsistency> detectMailboxDaoInconsistency(Mailbox mailbox) {
+ return mailboxPathV3DAO.retrieve(mailbox.generateAssociatedPath())
+ .map(mailboxByPath -> {
+ if (mailboxByPath.getMailboxId().equals(mailbox.getMailboxId())) {
return NO_INCONSISTENCY;
}
// Path entry references another mailbox.
- return new ConflictingEntryInconsistency(mailbox, pathRegistration);
+ return new ConflictingEntryInconsistency(ConflictingEntry.builder()
+ .mailboxDaoEntry(mailbox)
+ .mailboxPathDaoEntry(mailboxByPath));
})
.defaultIfEmpty(new OrphanMailboxDAOEntry(mailbox));
}
- private Mono<Inconsistency> detectInconsistency(CassandraIdAndPath pathRegistration) {
- return mailboxDAO.retrieveMailbox(pathRegistration.getCassandraId())
- .map(mailbox -> {
- if (mailbox.generateAssociatedPath().equals(pathRegistration.getMailboxPath())) {
+ private Mono<Inconsistency> detectMailboxPathDaoInconsistency(Mailbox mailboxByPath) {
+ CassandraId cassandraId = (CassandraId) mailboxByPath.getMailboxId();
+
+ return mailboxDAO.retrieveMailbox(cassandraId)
+ .map(mailboxById -> {
+ if (mailboxByPath.generateAssociatedPath().equals(mailboxById.generateAssociatedPath())) {
return NO_INCONSISTENCY;
}
// Mailbox references another path
- return new ConflictingEntryInconsistency(mailbox, pathRegistration);
+ return new ConflictingEntryInconsistency(ConflictingEntry.builder()
+ .mailboxDaoEntry(mailboxById)
+ .mailboxPathDaoEntry(mailboxByPath));
})
- .defaultIfEmpty(new OrphanMailboxPathDAOEntry(pathRegistration));
+ .defaultIfEmpty(new OrphanMailboxPathDAOEntry(mailboxByPath));
}
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesServiceTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesServiceTest.java
index 623d4a3..4bf8ec1 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesServiceTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesServiceTest.java
@@ -33,9 +33,8 @@ import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule
import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.core.Username;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
-import org.apache.james.mailbox.cassandra.mail.CassandraIdAndPath;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
-import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesService.Context;
import org.apache.james.mailbox.cassandra.modules.CassandraAclModule;
import org.apache.james.mailbox.cassandra.modules.CassandraMailboxModule;
@@ -54,9 +53,11 @@ class SolveMailboxInconsistenciesServiceTest {
private static final Username USER = Username.of("user");
private static final MailboxPath MAILBOX_PATH = MailboxPath.forUser(USER, "abc");
private static final MailboxPath NEW_MAILBOX_PATH = MailboxPath.forUser(USER, "xyz");
- private static CassandraId CASSANDRA_ID_1 = CassandraId.timeBased();
+ private static final CassandraId CASSANDRA_ID_1 = CassandraId.timeBased();
private static final Mailbox MAILBOX = new Mailbox(MAILBOX_PATH, UID_VALIDITY_1, CASSANDRA_ID_1);
- private static CassandraId CASSANDRA_ID_2 = CassandraId.timeBased();
+ private static final Mailbox MAILBOX_NEW_PATH = new Mailbox(NEW_MAILBOX_PATH, UID_VALIDITY_1, CASSANDRA_ID_1);
+ private static final CassandraId CASSANDRA_ID_2 = CassandraId.timeBased();
+ private static final Mailbox MAILBOX_2 = new Mailbox(MAILBOX_PATH, UID_VALIDITY_1, CASSANDRA_ID_2);
@RegisterExtension
static CassandraClusterExtension cassandraCluster = new CassandraClusterExtension(
@@ -67,7 +68,7 @@ class SolveMailboxInconsistenciesServiceTest {
CassandraMailboxDAO mailboxDAO;
- CassandraMailboxPathV2DAO mailboxPathV2DAO;
+ CassandraMailboxPathV3DAO mailboxPathV3DAO;
CassandraSchemaVersionDAO versionDAO;
SolveMailboxInconsistenciesService testee;
@@ -77,14 +78,14 @@ class SolveMailboxInconsistenciesServiceTest {
cassandra.getConf(),
cassandra.getTypesProvider(),
cassandraCluster.getCassandraConsistenciesConfiguration());
- mailboxPathV2DAO = new CassandraMailboxPathV2DAO(
+ mailboxPathV3DAO = new CassandraMailboxPathV3DAO(
cassandra.getConf(),
CassandraUtils.WITH_DEFAULT_CONFIGURATION,
cassandraCluster.getCassandraConsistenciesConfiguration());
versionDAO = new CassandraSchemaVersionDAO(cassandra.getConf());
- testee = new SolveMailboxInconsistenciesService(mailboxDAO, mailboxPathV2DAO, new CassandraSchemaVersionManager(versionDAO));
+ testee = new SolveMailboxInconsistenciesService(mailboxDAO, mailboxPathV3DAO, new CassandraSchemaVersionManager(versionDAO));
- versionDAO.updateVersion(new SchemaVersion(7)).block();
+ versionDAO.updateVersion(new SchemaVersion(8)).block();
}
@Test
@@ -94,7 +95,7 @@ class SolveMailboxInconsistenciesServiceTest {
assertThatThrownBy(() -> testee.fixMailboxInconsistencies(new Context()).block())
.isInstanceOf(IllegalStateException.class)
- .hasMessage("Schema version 6 is required in order to ensure mailboxPathV2DAO to be correctly populated, got 5");
+ .hasMessage("Schema version 8 is required in order to ensure mailboxPathV3DAO to be correctly populated, got 5");
}
@Test
@@ -103,13 +104,13 @@ class SolveMailboxInconsistenciesServiceTest {
assertThatThrownBy(() -> testee.fixMailboxInconsistencies(new Context()).block())
.isInstanceOf(IllegalStateException.class)
- .hasMessage("Schema version 6 is required in order to ensure mailboxPathV2DAO to be correctly populated, got 5");
+ .hasMessage("Schema version 8 is required in order to ensure mailboxPathV3DAO to be correctly populated, got 5");
}
@Test
void fixMailboxInconsistenciesShouldNotFailWhenIsEqualToMailboxPathV2Migration() {
versionDAO.truncateVersion().block();
- versionDAO.updateVersion(new SchemaVersion(6)).block();
+ versionDAO.updateVersion(new SchemaVersion(8)).block();
assertThatCode(() -> testee.fixMailboxInconsistencies(new Context()).block())
.doesNotThrowAnyException();
@@ -118,7 +119,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldNotFailWhenIsAboveMailboxPathV2Migration() {
versionDAO.truncateVersion().block();
- versionDAO.updateVersion(new SchemaVersion(7)).block();
+ versionDAO.updateVersion(new SchemaVersion(8)).block();
assertThatCode(() -> testee.fixMailboxInconsistencies(new Context()).block())
.doesNotThrowAnyException();
@@ -133,7 +134,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldReturnCompletedWhenConsistentData() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
assertThat(testee.fixMailboxInconsistencies(new Context()).block())
.isEqualTo(Result.COMPLETED);
@@ -149,7 +150,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldReturnCompletedWhenOrphanPathData() {
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
assertThat(testee.fixMailboxInconsistencies(new Context()).block())
.isEqualTo(Result.COMPLETED);
@@ -158,7 +159,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldReturnPartialWhenDAOMisMatchOnId() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
assertThat(testee.fixMailboxInconsistencies(new Context()).block())
.isEqualTo(Result.PARTIAL);
@@ -167,7 +168,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldReturnPartialWhenDAOMisMatchOnPath() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(NEW_MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX_NEW_PATH).block();
assertThat(testee.fixMailboxInconsistencies(new Context()).block())
.isEqualTo(Result.PARTIAL);
@@ -186,7 +187,7 @@ class SolveMailboxInconsistenciesServiceTest {
void fixMailboxInconsistenciesShouldUpdateContextWhenConsistentData() {
Context context = new Context();
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
testee.fixMailboxInconsistencies(context).block();
@@ -217,7 +218,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void fixMailboxInconsistenciesShouldUpdateContextWhenOrphanPathData() {
Context context = new Context();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
testee.fixMailboxInconsistencies(context).block();
@@ -233,7 +234,7 @@ class SolveMailboxInconsistenciesServiceTest {
void fixMailboxInconsistenciesShouldUpdateContextWhenDAOMisMatchOnId() {
Context context = new Context();
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
mailboxDAO.save(new Mailbox(MAILBOX_PATH, UID_VALIDITY_2, CASSANDRA_ID_2)).block();
testee.fixMailboxInconsistencies(context).block();
@@ -253,7 +254,7 @@ class SolveMailboxInconsistenciesServiceTest {
void fixMailboxInconsistenciesShouldUpdateContextWhenDAOMisMatchOnPath() {
Context context = new Context();
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(NEW_MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX_NEW_PATH).block();
testee.fixMailboxInconsistencies(context).block();
@@ -264,7 +265,7 @@ class SolveMailboxInconsistenciesServiceTest {
.addFixedInconsistencies(CASSANDRA_ID_1)
.addConflictingEntry(ConflictingEntry.builder()
.mailboxDaoEntry(MAILBOX)
- .mailboxPathDaoEntry(NEW_MAILBOX_PATH, CASSANDRA_ID_1))
+ .mailboxPathDaoEntry(MAILBOX_NEW_PATH))
.build()
.snapshot());
}
@@ -275,22 +276,22 @@ class SolveMailboxInconsistenciesServiceTest {
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block()).isEmpty();
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block()).isEmpty();
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block()).isEmpty();
});
}
@Test
void fixMailboxInconsistenciesShouldNotAlterStateWhenConsistent() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
testee.fixMailboxInconsistencies(new Context()).block();
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
- .containsExactlyInAnyOrder(new CassandraIdAndPath(CASSANDRA_ID_1, MAILBOX_PATH));
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX);
});
}
@@ -303,21 +304,21 @@ class SolveMailboxInconsistenciesServiceTest {
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
- .containsExactlyInAnyOrder(new CassandraIdAndPath(CASSANDRA_ID_1, MAILBOX_PATH));
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX);
});
}
@Test
void fixMailboxInconsistenciesShouldAlterStateWhenOrphanMailboxPath() {
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
testee.fixMailboxInconsistencies(new Context()).block();
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.isEmpty();
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
.isEmpty();
});
}
@@ -327,18 +328,18 @@ class SolveMailboxInconsistenciesServiceTest {
mailboxDAO.save(MAILBOX).block();
Mailbox mailbox2 = new Mailbox(NEW_MAILBOX_PATH, UID_VALIDITY_2, CASSANDRA_ID_2);
mailboxDAO.save(mailbox2).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
- mailboxPathV2DAO.save(NEW_MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
+ mailboxPathV3DAO.save(MAILBOX_NEW_PATH).block();
testee.fixMailboxInconsistencies(new Context()).block();
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX, mailbox2);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
.containsExactlyInAnyOrder(
- new CassandraIdAndPath(CASSANDRA_ID_1, NEW_MAILBOX_PATH),
- new CassandraIdAndPath(CASSANDRA_ID_2, MAILBOX_PATH));
+ MAILBOX_NEW_PATH,
+ MAILBOX_2);
});
}
@@ -347,31 +348,31 @@ class SolveMailboxInconsistenciesServiceTest {
// Note that CASSANDRA_ID_1 becomes usable
// However in order to avoid data loss, merging CASSANDRA_ID_1 and CASSANDRA_ID_2 is still required
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(NEW_MAILBOX_PATH, CASSANDRA_ID_1).block();
+ mailboxPathV3DAO.save(MAILBOX_NEW_PATH).block();
testee.fixMailboxInconsistencies(new Context()).block();
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
.containsExactlyInAnyOrder(
- new CassandraIdAndPath(CASSANDRA_ID_1, NEW_MAILBOX_PATH),
- new CassandraIdAndPath(CASSANDRA_ID_1, MAILBOX_PATH));
+ MAILBOX_NEW_PATH,
+ MAILBOX);
});
}
@Test
void fixMailboxInconsistenciesShouldAlterStateWhenDaoMisMatchOnId() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
testee.fixMailboxInconsistencies(new Context()).block();
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
.isEmpty();
});
}
@@ -379,7 +380,7 @@ class SolveMailboxInconsistenciesServiceTest {
@Test
void multipleRunShouldDaoMisMatchOnId() {
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
testee.fixMailboxInconsistencies(new Context()).block();
testee.fixMailboxInconsistencies(new Context()).block();
@@ -387,8 +388,8 @@ class SolveMailboxInconsistenciesServiceTest {
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
- .containsExactlyInAnyOrder(new CassandraIdAndPath(CASSANDRA_ID_1, MAILBOX_PATH));
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX);
});
}
@@ -398,7 +399,7 @@ class SolveMailboxInconsistenciesServiceTest {
Mailbox mailbox2 = new Mailbox(MAILBOX_PATH, UID_VALIDITY_2, CASSANDRA_ID_2);
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, CASSANDRA_ID_2).block();
+ mailboxPathV3DAO.save(MAILBOX_2).block();
mailboxDAO.save(mailbox2).block();
testee.fixMailboxInconsistencies(new Context()).block();
@@ -406,9 +407,8 @@ class SolveMailboxInconsistenciesServiceTest {
SoftAssertions.assertSoftly(softly -> {
softly.assertThat(mailboxDAO.retrieveAllMailboxes().collectList().block())
.containsExactlyInAnyOrder(MAILBOX, mailbox2);
- softly.assertThat(mailboxPathV2DAO.listAll().collectList().block())
- .containsExactlyInAnyOrder(
- new CassandraIdAndPath(CASSANDRA_ID_2, MAILBOX_PATH));
+ softly.assertThat(mailboxPathV3DAO.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX_2);
});
}
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 17/22: JAMES-3409 FIX
CassandraMailboxManagerConsistencyTest
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 0ae02fa4041443b2907e2b6fd3f0b4479542ccc6
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue Aug 18 15:59:48 2020 +0700
JAMES-3409 FIX CassandraMailboxManagerConsistencyTest
---
.../mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
index ec299f9..5108b62 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
@@ -79,7 +79,7 @@ class CassandraMailboxManagerConsistencyTest {
@Nested
class FailuresDuringCreation {
- @Disabled("oups")
+ @Disabled("For performance reasons we don't validate path reads against mailbox table")
@Test
void createMailboxShouldBeConsistentWhenMailboxDaoFails(CassandraCluster cassandra) {
cassandra.getConf().registerScenario(fail()
@@ -200,7 +200,7 @@ class CassandraMailboxManagerConsistencyTest {
@Nested
class FailuresDuringRenaming {
- @Disabled("oups")
+ @Disabled("For performance reasons we don't validate path reads against mailbox table")
@Test
void renameShouldBeConsistentWhenMailboxDaoFails(CassandraCluster cassandra) throws Exception {
MailboxId inboxId = testee.createMailbox(inboxPath, mailboxSession)
@@ -633,7 +633,7 @@ class CassandraMailboxManagerConsistencyTest {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3"));
doQuietly(() -> testee.deleteMailbox(inboxPath, mailboxSession));
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 14/22: JAMES-3409 Fix FixingGhostMailboxTest
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 44f50c2348f4abce8e9a3ad46e021bdd5f0a1601
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 31 09:37:51 2020 +0700
JAMES-3409 Fix FixingGhostMailboxTest
---
.../webadmin/integration/rabbitmq/FixingGhostMailboxTest.java | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/FixingGhostMailboxTest.java b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/FixingGhostMailboxTest.java
index f370286..be3e26b 100644
--- a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/FixingGhostMailboxTest.java
+++ b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/FixingGhostMailboxTest.java
@@ -59,7 +59,7 @@ import org.apache.james.jmap.draft.JmapGuiceProbe;
import org.apache.james.junit.categories.BasicFeature;
import org.apache.james.mailbox.MessageManager.AppendCommand;
import org.apache.james.mailbox.cassandra.mail.task.MailboxMergingTask;
-import org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV2Table;
+import org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.model.ComposedMessageId;
import org.apache.james.mailbox.model.MailboxACL;
@@ -182,10 +182,10 @@ class FixingGhostMailboxTest {
testExtension.await();
// Simulate ghost mailbox bug
- session.execute(delete().from(CassandraMailboxPathV2Table.TABLE_NAME)
- .where(eq(CassandraMailboxPathV2Table.NAMESPACE, MailboxConstants.USER_NAMESPACE))
- .and(eq(CassandraMailboxPathV2Table.USER, ALICE))
- .and(eq(CassandraMailboxPathV2Table.MAILBOX_NAME, MailboxConstants.INBOX)));
+ session.execute(delete().from(CassandraMailboxPathV3Table.TABLE_NAME)
+ .where(eq(CassandraMailboxPathV3Table.NAMESPACE, MailboxConstants.USER_NAMESPACE))
+ .and(eq(CassandraMailboxPathV3Table.USER, ALICE))
+ .and(eq(CassandraMailboxPathV3Table.MAILBOX_NAME, MailboxConstants.INBOX)));
// trigger provisioning
given()
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 10/22: JAMES-3409 CassandraMailboxMapper should
rely on MailboxPath V3 DAO
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit b4a0e8f25ff5e8c36c58c2cf8683a56b94bf16e3
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Jul 29 17:23:00 2020 +0700
JAMES-3409 CassandraMailboxMapper should rely on MailboxPath V3 DAO
---
.../CassandraMailboxSessionMapperFactory.java | 7 +-
.../cassandra/mail/CassandraMailboxMapper.java | 84 ++++++++------
.../CassandraMailboxManagerConsistencyTest.java | 10 +-
.../CassandraSubscriptionManagerTest.java | 3 +
.../cassandra/mail/CassandraMailboxMapperTest.java | 129 +++++++++++++++++++--
.../mail/migration/MailboxPathV2MigrationTest.java | 51 ++++++--
6 files changed, 225 insertions(+), 59 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
index e51364c..9767519 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/CassandraMailboxSessionMapperFactory.java
@@ -41,6 +41,7 @@ import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxMapper;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathDAOImpl;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxRecentsDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
@@ -81,6 +82,7 @@ public class CassandraMailboxSessionMapperFactory extends MailboxSessionMapperFa
private final CassandraMailboxDAO mailboxDAO;
private final CassandraMailboxPathDAOImpl mailboxPathDAO;
private final CassandraMailboxPathV2DAO mailboxPathV2DAO;
+ private final CassandraMailboxPathV3DAO mailboxPathV3DAO;
private final CassandraFirstUnseenDAO firstUnseenDAO;
private final CassandraApplicableFlagDAO applicableFlagDAO;
private final CassandraAttachmentDAOV2 attachmentDAOV2;
@@ -99,7 +101,7 @@ public class CassandraMailboxSessionMapperFactory extends MailboxSessionMapperFa
CassandraMessageDAO messageDAO,
CassandraMessageIdDAO messageIdDAO, CassandraMessageIdToImapUidDAO imapUidDAO,
CassandraMailboxCounterDAO mailboxCounterDAO, CassandraMailboxRecentsDAO mailboxRecentsDAO, CassandraMailboxDAO mailboxDAO,
- CassandraMailboxPathDAOImpl mailboxPathDAO, CassandraMailboxPathV2DAO mailboxPathV2DAO, CassandraFirstUnseenDAO firstUnseenDAO, CassandraApplicableFlagDAO applicableFlagDAO,
+ CassandraMailboxPathDAOImpl mailboxPathDAO, CassandraMailboxPathV2DAO mailboxPathV2DAO, CassandraMailboxPathV3DAO mailboxPathV3DAO, CassandraFirstUnseenDAO firstUnseenDAO, CassandraApplicableFlagDAO applicableFlagDAO,
CassandraAttachmentDAOV2 attachmentDAOV2, CassandraDeletedMessageDAO deletedMessageDAO,
BlobStore blobStore, CassandraAttachmentMessageIdDAO attachmentMessageIdDAO,
CassandraAttachmentOwnerDAO ownerDAO, CassandraACLMapper aclMapper,
@@ -117,6 +119,7 @@ public class CassandraMailboxSessionMapperFactory extends MailboxSessionMapperFa
this.mailboxDAO = mailboxDAO;
this.mailboxPathDAO = mailboxPathDAO;
this.mailboxPathV2DAO = mailboxPathV2DAO;
+ this.mailboxPathV3DAO = mailboxPathV3DAO;
this.firstUnseenDAO = firstUnseenDAO;
this.attachmentDAOV2 = attachmentDAOV2;
this.deletedMessageDAO = deletedMessageDAO;
@@ -165,7 +168,7 @@ public class CassandraMailboxSessionMapperFactory extends MailboxSessionMapperFa
@Override
public MailboxMapper createMailboxMapper(MailboxSession mailboxSession) {
- return new CassandraMailboxMapper(mailboxDAO, mailboxPathDAO, mailboxPathV2DAO, userMailboxRightsDAO, aclMapper, versionManager);
+ return new CassandraMailboxMapper(mailboxDAO, mailboxPathDAO, mailboxPathV2DAO, mailboxPathV3DAO, userMailboxRightsDAO, aclMapper, versionManager);
}
@Override
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
index 0e13275..84b5b24 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
@@ -57,11 +57,12 @@ public class CassandraMailboxMapper implements MailboxMapper {
private static final int MAX_RETRY = 5;
private static final Duration MIN_RETRY_BACKOFF = Duration.ofMillis(10);
private static final Duration MAX_RETRY_BACKOFF = Duration.ofMillis(1000);
- private static final SchemaVersion MAILBOX_PATH_V_2_MIGRATION_PERFORMED_VERSION = new SchemaVersion(6);
+ private static final SchemaVersion MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION = new SchemaVersion(8);
private final CassandraMailboxDAO mailboxDAO;
private final CassandraMailboxPathDAOImpl mailboxPathDAO;
private final CassandraMailboxPathV2DAO mailboxPathV2DAO;
+ private final CassandraMailboxPathV3DAO mailboxPathV3DAO;
private final CassandraACLMapper cassandraACLMapper;
private final CassandraUserMailboxRightsDAO userMailboxRightsDAO;
private final CassandraSchemaVersionManager versionManager;
@@ -70,19 +71,21 @@ public class CassandraMailboxMapper implements MailboxMapper {
public CassandraMailboxMapper(CassandraMailboxDAO mailboxDAO,
CassandraMailboxPathDAOImpl mailboxPathDAO,
CassandraMailboxPathV2DAO mailboxPathV2DAO,
+ CassandraMailboxPathV3DAO mailboxPathV3DAO,
CassandraUserMailboxRightsDAO userMailboxRightsDAO,
CassandraACLMapper aclMapper,
CassandraSchemaVersionManager versionManager) {
this.mailboxDAO = mailboxDAO;
this.mailboxPathDAO = mailboxPathDAO;
this.mailboxPathV2DAO = mailboxPathV2DAO;
+ this.mailboxPathV3DAO = mailboxPathV3DAO;
this.userMailboxRightsDAO = userMailboxRightsDAO;
this.cassandraACLMapper = aclMapper;
this.versionManager = versionManager;
}
- private Mono<Boolean> needMailboxPathV1Support() {
- return versionManager.isBefore(MAILBOX_PATH_V_2_MIGRATION_PERFORMED_VERSION);
+ private Mono<Boolean> needMailboxPathPreviousVersionsSupport() {
+ return versionManager.isBefore(MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION);
}
@Override
@@ -94,51 +97,62 @@ public class CassandraMailboxMapper implements MailboxMapper {
}
private Flux<Void> deletePath(Mailbox mailbox) {
- return needMailboxPathV1Support()
+ return needMailboxPathPreviousVersionsSupport()
.flatMapMany(needSupport -> {
if (needSupport) {
return Flux.merge(
mailboxPathDAO.delete(mailbox.generateAssociatedPath()),
- mailboxPathV2DAO.delete(mailbox.generateAssociatedPath()));
+ mailboxPathV2DAO.delete(mailbox.generateAssociatedPath()),
+ mailboxPathV3DAO.delete(mailbox.generateAssociatedPath()));
}
- return Flux.from(mailboxPathV2DAO.delete(mailbox.generateAssociatedPath()));
+ return Flux.from(mailboxPathV3DAO.delete(mailbox.generateAssociatedPath()));
});
}
@Override
public Mono<Mailbox> findMailboxByPath(MailboxPath path) {
- return mailboxPathV2DAO.retrieveId(path)
- .map(CassandraIdAndPath::getCassandraId)
- .flatMap(this::retrieveMailbox)
- .switchIfEmpty(fromPreviousTable(path));
+ return mailboxPathV3DAO.retrieve(path)
+ .switchIfEmpty(fromPreviousTable(path))
+ .flatMap(this::addAcl);
+ }
+
+ private Mono<Mailbox> addAcl(Mailbox mailbox) {
+ CassandraId mailboxId = (CassandraId) mailbox.getMailboxId();
+ return cassandraACLMapper.getACL(mailboxId)
+ .map(acl -> {
+ mailbox.setACL(acl);
+ return mailbox;
+ })
+ .switchIfEmpty(Mono.just(mailbox));
}
@Override
public Mono<Boolean> pathExists(MailboxPath mailboxName) {
- return mailboxPathV2DAO.retrieveId(mailboxName)
- .switchIfEmpty(mailboxPathDAO.retrieveId(mailboxName))
+ return mailboxPathV3DAO.retrieve(mailboxName)
+ .switchIfEmpty(fromPreviousTable(mailboxName))
.hasElement();
}
private Mono<Mailbox> fromPreviousTable(MailboxPath path) {
- return mailboxPathDAO.retrieveId(path)
+ return mailboxPathV2DAO.retrieveId(path)
+ .switchIfEmpty(mailboxPathDAO.retrieveId(path))
.map(CassandraIdAndPath::getCassandraId)
.flatMap(this::retrieveMailbox)
.flatMap(this::migrate);
}
private Mono<Mailbox> migrate(Mailbox mailbox) {
- CassandraId mailboxId = (CassandraId) mailbox.getMailboxId();
- return mailboxPathV2DAO.save(mailbox.generateAssociatedPath(), mailboxId)
+ return mailboxPathV3DAO.save(mailbox)
.flatMap(success -> deleteIfSuccess(mailbox, success))
.thenReturn(mailbox);
}
private Mono<Void> deleteIfSuccess(Mailbox mailbox, boolean success) {
if (success) {
- return mailboxPathDAO.delete(mailbox.generateAssociatedPath());
+ return mailboxPathDAO.delete(mailbox.generateAssociatedPath())
+ .then(mailboxPathV2DAO.delete(mailbox.generateAssociatedPath()));
}
- LOGGER.info("Concurrent execution lead to data race while migrating {} to 'mailboxPathV2DAO'.",
+ LOGGER.info("Concurrent execution lead to data race while migrating {} to 'mailboxPathV3DAO'.",
mailbox.generateAssociatedPath());
return Mono.empty();
}
@@ -172,20 +186,24 @@ public class CassandraMailboxMapper implements MailboxMapper {
String fixedNamespace = query.getFixedNamespace();
Username fixedUser = query.getFixedUser();
- return listPaths(fixedNamespace, fixedUser)
- .filter(idAndPath -> query.isPathMatch(idAndPath.getMailboxPath()))
- .distinct(CassandraIdAndPath::getMailboxPath)
- .concatMap(this::retrieveMailbox);
+ return listMailboxes(fixedNamespace, fixedUser)
+ .filter(mailbox -> query.isPathMatch(mailbox.generateAssociatedPath()))
+ .distinct(Mailbox::generateAssociatedPath)
+ .flatMap(this::addAcl);
}
- private Flux<CassandraIdAndPath> listPaths(String fixedNamespace, Username fixedUser) {
- return needMailboxPathV1Support()
+ private Flux<Mailbox> listMailboxes(String fixedNamespace, Username fixedUser) {
+ return needMailboxPathPreviousVersionsSupport()
.flatMapMany(needSupport -> {
if (needSupport) {
- return Flux.concat(mailboxPathV2DAO.listUserMailboxes(fixedNamespace, fixedUser),
- mailboxPathDAO.listUserMailboxes(fixedNamespace, fixedUser));
+ return Flux.concat(
+ mailboxPathV3DAO.listUserMailboxes(fixedNamespace, fixedUser),
+ Flux.concat(
+ mailboxPathV2DAO.listUserMailboxes(fixedNamespace, fixedUser),
+ mailboxPathDAO.listUserMailboxes(fixedNamespace, fixedUser))
+ .flatMap(this::retrieveMailbox));
}
- return mailboxPathV2DAO.listUserMailboxes(fixedNamespace, fixedUser);
+ return mailboxPathV3DAO.listUserMailboxes(fixedNamespace, fixedUser);
});
}
@@ -200,7 +218,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
CassandraId cassandraId = CassandraId.timeBased();
Mailbox mailbox = new Mailbox(mailboxPath, uidValidity, cassandraId);
- return mailboxPathV2DAO.save(mailbox.generateAssociatedPath(), cassandraId)
+ return mailboxPathV3DAO.save(mailbox)
.filter(isCreated -> isCreated)
.flatMap(mailboxHasCreated -> persistMailboxEntity(mailbox)
.thenReturn(mailbox))
@@ -220,7 +238,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
private Mono<Boolean> tryRename(Mailbox cassandraMailbox, CassandraId cassandraId) {
return mailboxDAO.retrieveMailbox(cassandraId)
- .flatMap(mailbox -> mailboxPathV2DAO.save(cassandraMailbox.generateAssociatedPath(), cassandraId)
+ .flatMap(mailbox -> mailboxPathV3DAO.save(cassandraMailbox)
.filter(isCreated -> isCreated)
.flatMap(mailboxHasCreated -> deletePreviousMailboxPathReference(mailbox.generateAssociatedPath())
.then(persistMailboxEntity(cassandraMailbox))
@@ -235,21 +253,19 @@ public class CassandraMailboxMapper implements MailboxMapper {
}
private Mono<Void> deletePreviousMailboxPathReference(MailboxPath mailboxPath) {
- return mailboxPathV2DAO.delete(mailboxPath)
+ return mailboxPathV3DAO.delete(mailboxPath)
.retryWhen(Retry.backoff(MAX_RETRY, MIN_RETRY_BACKOFF).maxBackoff(MAX_RETRY_BACKOFF));
}
@Override
public Mono<Boolean> hasChildren(Mailbox mailbox, char delimiter) {
- return Flux.merge(
- mailboxPathDAO.listUserMailboxes(mailbox.getNamespace(), mailbox.getUser()),
- mailboxPathV2DAO.listUserMailboxes(mailbox.getNamespace(), mailbox.getUser()))
+ return listMailboxes(mailbox.getNamespace(), mailbox.getUser())
.filter(idAndPath -> isPathChildOfMailbox(idAndPath, mailbox, delimiter))
.hasElements();
}
- private boolean isPathChildOfMailbox(CassandraIdAndPath idAndPath, Mailbox mailbox, char delimiter) {
- return idAndPath.getMailboxPath().getName().startsWith(mailbox.getName() + String.valueOf(delimiter));
+ private boolean isPathChildOfMailbox(Mailbox candidate, Mailbox mailbox, char delimiter) {
+ return candidate.generateAssociatedPath().getName().startsWith(mailbox.getName() + delimiter);
}
@Override
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
index 345a5e7..a81c8c6 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraMailboxManagerConsistencyTest.java
@@ -79,6 +79,7 @@ class CassandraMailboxManagerConsistencyTest {
@Nested
class FailuresDuringCreation {
+ @Disabled("oups")
@Test
void createMailboxShouldBeConsistentWhenMailboxDaoFails(CassandraCluster cassandra) {
cassandra.getConf().registerScenario(fail()
@@ -101,7 +102,7 @@ class CassandraMailboxManagerConsistencyTest {
void createMailboxShouldBeConsistentWhenMailboxPathDaoFails(CassandraCluster cassandra) {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("INSERT INTO mailboxPathV2"));
+ .whenQueryStartsWith("INSERT INTO mailboxPathV3"));
doQuietly(() -> testee.createMailbox(inboxPath, mailboxSession));
@@ -132,7 +133,7 @@ class CassandraMailboxManagerConsistencyTest {
void createMailboxAfterAFailedCreationShouldCreateTheMailboxWhenMailboxPathDaoFails(CassandraCluster cassandra) throws Exception {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("INSERT INTO mailboxPathV2"));
+ .whenQueryStartsWith("INSERT INTO mailboxPathV3"));
doQuietly(() -> testee.createMailbox(inboxPath, mailboxSession));
@@ -199,6 +200,7 @@ class CassandraMailboxManagerConsistencyTest {
@Nested
class FailuresDuringRenaming {
+ @Disabled("oups")
@Test
void renameShouldBeConsistentWhenMailboxDaoFails(CassandraCluster cassandra) throws Exception {
MailboxId inboxId = testee.createMailbox(inboxPath, mailboxSession)
@@ -230,7 +232,7 @@ class CassandraMailboxManagerConsistencyTest {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("INSERT INTO mailboxPathV2"));
+ .whenQueryStartsWith("INSERT INTO mailboxPathV3"));
doQuietly(() -> testee.renameMailbox(inboxPath, inboxPathRenamed, mailboxSession));
@@ -287,7 +289,7 @@ class CassandraMailboxManagerConsistencyTest {
cassandra.getConf().registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("INSERT INTO mailboxPathV2"));
+ .whenQueryStartsWith("INSERT INTO mailboxPathV3"));
doQuietly(() -> testee.renameMailbox(inboxPath, inboxPathRenamed, mailboxSession));
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraSubscriptionManagerTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraSubscriptionManagerTest.java
index f555a7f..781ef96 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraSubscriptionManagerTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/CassandraSubscriptionManagerTest.java
@@ -37,6 +37,7 @@ import org.apache.james.mailbox.cassandra.mail.CassandraMailboxCounterDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathDAOImpl;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxRecentsDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
@@ -74,6 +75,7 @@ class CassandraSubscriptionManagerTest implements SubscriptionManagerContract {
CassandraMailboxDAO mailboxDAO = null;
CassandraMailboxPathDAOImpl mailboxPathDAO = null;
CassandraMailboxPathV2DAO mailboxPathV2DAO = null;
+ CassandraMailboxPathV3DAO mailboxPathV3DAO = null;
CassandraFirstUnseenDAO firstUnseenDAO = null;
CassandraApplicableFlagDAO applicableFlagDAO = null;
CassandraDeletedMessageDAO deletedMessageDAO = null;
@@ -100,6 +102,7 @@ class CassandraSubscriptionManagerTest implements SubscriptionManagerContract {
mailboxDAO,
mailboxPathDAO,
mailboxPathV2DAO,
+ mailboxPathV3DAO,
firstUnseenDAO,
applicableFlagDAO,
attachmentDAOV2,
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
index ea6ce6b..0066a40 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
@@ -36,6 +36,7 @@ import org.apache.james.backends.cassandra.utils.CassandraUtils;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionDAO;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionManager;
import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule;
+import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.core.Username;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
import org.apache.james.mailbox.cassandra.modules.CassandraAclModule;
@@ -84,7 +85,9 @@ class CassandraMailboxMapperTest {
private CassandraMailboxDAO mailboxDAO;
private CassandraMailboxPathDAOImpl mailboxPathDAO;
private CassandraMailboxPathV2DAO mailboxPathV2DAO;
+ private CassandraMailboxPathV3DAO mailboxPathV3DAO;
private CassandraMailboxMapper testee;
+ private CassandraSchemaVersionDAO versionDAO;
@BeforeEach
void setUp() {
@@ -92,19 +95,22 @@ class CassandraMailboxMapperTest {
mailboxDAO = new CassandraMailboxDAO(cassandra.getConf(), cassandra.getTypesProvider(), cassandraCluster.getCassandraConsistenciesConfiguration());
mailboxPathDAO = new CassandraMailboxPathDAOImpl(cassandra.getConf(), cassandra.getTypesProvider(), cassandraCluster.getCassandraConsistenciesConfiguration());
mailboxPathV2DAO = new CassandraMailboxPathV2DAO(cassandra.getConf(), CassandraUtils.WITH_DEFAULT_CONFIGURATION, cassandraCluster.getCassandraConsistenciesConfiguration());
+ mailboxPathV3DAO = new CassandraMailboxPathV3DAO(cassandra.getConf(), CassandraUtils.WITH_DEFAULT_CONFIGURATION, cassandraCluster.getCassandraConsistenciesConfiguration());
CassandraUserMailboxRightsDAO userMailboxRightsDAO = new CassandraUserMailboxRightsDAO(cassandra.getConf(), CassandraUtils.WITH_DEFAULT_CONFIGURATION);
CassandraACLMapper aclMapper = new CassandraACLMapper(
cassandra.getConf(),
new CassandraUserMailboxRightsDAO(cassandra.getConf(), CassandraUtils.WITH_DEFAULT_CONFIGURATION),
CassandraConfiguration.DEFAULT_CONFIGURATION,
cassandraCluster.getCassandraConsistenciesConfiguration());
+ versionDAO = new CassandraSchemaVersionDAO(cassandra.getConf());
testee = new CassandraMailboxMapper(
mailboxDAO,
mailboxPathDAO,
mailboxPathV2DAO,
+ mailboxPathV3DAO,
userMailboxRightsDAO,
aclMapper,
- new CassandraSchemaVersionManager(new CassandraSchemaVersionDAO(cassandra.getConf())));
+ new CassandraSchemaVersionManager(versionDAO));
}
@Nested
@@ -242,6 +248,9 @@ class CassandraMailboxMapperTest {
}
}
+ @Disabled("In order to be more performant mailboxPath V3 table includes the UID_VALIDITY." +
+ "Reading paths no longer requires reading the mailbox by id but this of course has a " +
+ "consistency cost.")
@Test
void createShouldBeConsistentWhenFailToPersistMailbox(CassandraCluster cassandra) {
cassandra.getConf()
@@ -348,7 +357,7 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
doQuietly(() -> testee.rename(inboxRenamed).block());
@@ -377,7 +386,7 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
doQuietly(() -> testee.rename(inboxRenamed).block());
@@ -399,7 +408,7 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
doQuietly(() -> testee.rename(inboxRenamed).block());
@@ -450,7 +459,7 @@ class CassandraMailboxMapperTest {
CassandraId inboxId = (CassandraId) inbox.getMailboxId();
// simulate mailbox old data has not been migrated to v2
mailboxPathDAO.save(inboxPath, inboxId).block();
- mailboxPathV2DAO.delete(inboxPath).block();
+ mailboxPathV3DAO.delete(inboxPath).block();
// on current v2 generation, save a new mailbox with the exactly name
// => two mailboxes with same name but different ids
@@ -598,7 +607,7 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
.times(TRY_COUNT_BEFORE_FAILURE)
- .whenQueryStartsWith("DELETE FROM mailboxPathV2 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
+ .whenQueryStartsWith("DELETE FROM mailboxPathV3 WHERE namespace=:namespace AND user=:user AND mailboxName=:mailboxName IF EXISTS;"));
doQuietly(() -> testee.rename(inboxRenamed).block());
@@ -651,7 +660,7 @@ class CassandraMailboxMapperTest {
assertThatThrownBy(() -> testee.rename(newMailbox).block())
.isInstanceOf(TooLongMailboxNameException.class);
- assertThat(mailboxPathV2DAO.retrieveId(MAILBOX_PATH).blockOptional())
+ assertThat(mailboxPathV3DAO.retrieve(MAILBOX_PATH).blockOptional())
.isPresent();
}
@@ -686,6 +695,36 @@ class CassandraMailboxMapperTest {
}
@Test
+ void deleteShouldDeleteMailboxAndMailboxPathFromV3Table() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ testee.delete(MAILBOX).block();
+
+ assertThat(testee.findMailboxByPath(MAILBOX_PATH).blockOptional())
+ .isEmpty();
+ }
+
+ @Test
+ void deleteShouldDeleteMailboxAndMailboxPathFromAllTables() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+ mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
+ .block();
+ mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
+ .block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ testee.delete(MAILBOX).block();
+
+ assertThat(testee.findMailboxByPath(MAILBOX_PATH).blockOptional())
+ .isEmpty();
+ }
+
+ @Test
void findMailboxByPathShouldReturnMailboxWhenExistsInV1Table() {
mailboxDAO.save(MAILBOX)
.block();
@@ -710,13 +749,27 @@ class CassandraMailboxMapperTest {
}
@Test
- void findMailboxByPathShouldReturnMailboxWhenExistsInBothTables() {
+ void findMailboxByPathShouldReturnMailboxWhenExistsInV3Table() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ Mailbox mailbox = testee.findMailboxByPath(MAILBOX_PATH).block();
+
+ assertThat(mailbox.generateAssociatedPath()).isEqualTo(MAILBOX_PATH);
+ }
+
+ @Test
+ void findMailboxByPathShouldReturnMailboxWhenExistsInAllTables() {
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
.block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
Mailbox mailbox = testee.findMailboxByPath(MAILBOX_PATH).block();
@@ -724,13 +777,15 @@ class CassandraMailboxMapperTest {
}
@Test
- void deleteShouldRemoveMailboxWhenInBothTables() {
+ void deleteShouldRemoveMailboxWhenInAllTables() {
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
.block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
testee.delete(MAILBOX).block();
@@ -775,6 +830,8 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInV1Table() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -793,6 +850,8 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInBothTables() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -813,6 +872,8 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInV2Table() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -830,7 +891,28 @@ class CassandraMailboxMapperTest {
}
@Test
+ void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInV3Table() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ List<Mailbox> mailboxes = testee.findMailboxWithPathLike(MailboxQuery.builder()
+ .privateNamespace()
+ .username(USER)
+ .expression(Wildcard.INSTANCE)
+ .build()
+ .asUserBound())
+ .collectList()
+ .block();
+
+ assertThat(mailboxes).containsOnly(MAILBOX);
+ }
+
+ @Test
void hasChildrenShouldReturnChildWhenExistsInV1Table() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -850,6 +932,8 @@ class CassandraMailboxMapperTest {
@Test
void hasChildrenShouldReturnChildWhenExistsInBothTables() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -871,6 +955,8 @@ class CassandraMailboxMapperTest {
@Test
void hasChildrenShouldReturnChildWhenExistsInV2Table() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -889,9 +975,30 @@ class CassandraMailboxMapperTest {
}
@Test
- void findMailboxWithPathLikeShouldRemoveDuplicatesAndKeepV2() {
+ void hasChildrenShouldReturnChildWhenExistsInV3Table() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+ CassandraId childMailboxId = CassandraId.timeBased();
+ MailboxPath childMailboxPath = MailboxPath.forUser(USER, "name.child");
+ Mailbox childMailbox = new Mailbox(childMailboxPath, UID_VALIDITY, childMailboxId);
+ mailboxDAO.save(childMailbox)
+ .block();
+ mailboxPathV3DAO.save(childMailbox)
+ .block();
+
+ boolean hasChildren = testee.hasChildren(MAILBOX, '.').block();
+
+ assertThat(hasChildren).isTrue();
+ }
+
+ @Test
+ void findMailboxWithPathLikeShouldRemoveDuplicatesAndKeepV3() {
+ versionDAO.updateVersion(new SchemaVersion(7)).block();
+
mailboxDAO.save(MAILBOX).block();
- mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID).block();
+ mailboxPathV3DAO.save(MAILBOX).block();
mailboxDAO.save(MAILBOX_BIS).block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID_2).block();
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
index 6ea1b36..c0832ab 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV2MigrationTest.java
@@ -37,6 +37,7 @@ import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxMapper;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathDAOImpl;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.cassandra.mail.CassandraUserMailboxRightsDAO;
import org.apache.james.mailbox.cassandra.modules.CassandraAclModule;
import org.apache.james.mailbox.cassandra.modules.CassandraMailboxModule;
@@ -53,6 +54,8 @@ class MailboxPathV2MigrationTest {
private static final MailboxPath MAILBOX_PATH_1 = MailboxPath.forUser(Username.of("bob"), "Important");
private static final UidValidity UID_VALIDITY_1 = UidValidity.of(452);
private static final CassandraId MAILBOX_ID_1 = CassandraId.timeBased();
+ private static final Mailbox MAILBOX = new Mailbox(MAILBOX_PATH_1, UID_VALIDITY_1, MAILBOX_ID_1);
+
public static final CassandraModule MODULES = CassandraModule.aggregateModules(
CassandraMailboxModule.MODULE,
@@ -64,6 +67,7 @@ class MailboxPathV2MigrationTest {
private CassandraMailboxPathDAOImpl daoV1;
private CassandraMailboxPathV2DAO daoV2;
+ private CassandraMailboxPathV3DAO daoV3;
private CassandraMailboxMapper mailboxMapper;
private CassandraMailboxDAO mailboxDAO;
@@ -78,6 +82,10 @@ class MailboxPathV2MigrationTest {
cassandra.getConf(),
CassandraUtils.WITH_DEFAULT_CONFIGURATION,
cassandraCluster.getCassandraConsistenciesConfiguration());
+ daoV3 = new CassandraMailboxPathV3DAO(
+ cassandra.getConf(),
+ CassandraUtils.WITH_DEFAULT_CONFIGURATION,
+ cassandraCluster.getCassandraConsistenciesConfiguration());
CassandraUserMailboxRightsDAO userMailboxRightsDAO = new CassandraUserMailboxRightsDAO(cassandra.getConf(), CassandraUtils.WITH_DEFAULT_CONFIGURATION);
mailboxDAO = new CassandraMailboxDAO(
@@ -88,6 +96,7 @@ class MailboxPathV2MigrationTest {
mailboxDAO,
daoV1,
daoV2,
+ daoV3,
userMailboxRightsDAO,
new CassandraACLMapper(
cassandra.getConf(),
@@ -99,15 +108,16 @@ class MailboxPathV2MigrationTest {
@Test
void newValuesShouldBeSavedInMostRecentDAO() {
- Mailbox mailbox = createMailbox();
- CassandraId mailboxId = (CassandraId) mailbox.getMailboxId();
+ createMailbox();
- assertThat(daoV2.retrieveId(MAILBOX_PATH_1).blockOptional())
- .contains(new CassandraIdAndPath(mailboxId, MAILBOX_PATH_1));
+ assertThat(daoV3.retrieve(MAILBOX_PATH_1)
+ .map(Mailbox::generateAssociatedPath)
+ .blockOptional())
+ .contains(MAILBOX_PATH_1);
}
@Test
- void newValuesShouldNotBeSavedInOldDAO() {
+ void newValuesShouldNotBeSavedInV1DAO() {
createMailbox();
assertThat(daoV1.retrieveId(MAILBOX_PATH_1).blockOptional())
@@ -115,7 +125,16 @@ class MailboxPathV2MigrationTest {
}
@Test
- void readingOldValuesShouldMigrateThem() {
+ void newValuesShouldNotBeSavedInV2DAO() {
+ createMailbox();
+
+ assertThat(daoV2.retrieveId(MAILBOX_PATH_1)
+ .blockOptional())
+ .isEmpty();
+ }
+
+ @Test
+ void readingOldValuesShouldMigrateThemWhenV1() {
Mailbox mailbox = new Mailbox(MAILBOX_PATH_1, UID_VALIDITY_1, MAILBOX_ID_1);
daoV1.save(MAILBOX_PATH_1, MAILBOX_ID_1).block();
@@ -125,8 +144,24 @@ class MailboxPathV2MigrationTest {
SoftAssertions softly = new SoftAssertions();
softly.assertThat(daoV1.retrieveId(MAILBOX_PATH_1).blockOptional()).isEmpty();
- softly.assertThat(daoV2.retrieveId(MAILBOX_PATH_1).blockOptional())
- .contains(new CassandraIdAndPath(MAILBOX_ID_1, MAILBOX_PATH_1));
+ softly.assertThat(daoV3.retrieve(MAILBOX_PATH_1).blockOptional())
+ .contains(MAILBOX);
+ softly.assertAll();
+ }
+
+ @Test
+ void readingOldValuesShouldMigrateThemWhenV2() {
+ Mailbox mailbox = new Mailbox(MAILBOX_PATH_1, UID_VALIDITY_1, MAILBOX_ID_1);
+
+ daoV2.save(MAILBOX_PATH_1, MAILBOX_ID_1).block();
+ mailboxDAO.save(mailbox).block();
+
+ mailboxMapper.findMailboxByPath(MAILBOX_PATH_1).block();
+
+ SoftAssertions softly = new SoftAssertions();
+ softly.assertThat(daoV2.retrieveId(MAILBOX_PATH_1).blockOptional()).isEmpty();
+ softly.assertThat(daoV3.retrieve(MAILBOX_PATH_1).blockOptional())
+ .contains(MAILBOX);
softly.assertAll();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 18/22: JAMES-3409 FIX CassandraMailboxMapperTest
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4427954ba06dbf96f809e0f76a2128f6eea2f455
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Aug 19 09:32:29 2020 +0700
JAMES-3409 FIX CassandraMailboxMapperTest
---
.../cassandra/mail/CassandraMailboxMapperTest.java | 19 +++++--------------
1 file changed, 5 insertions(+), 14 deletions(-)
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
index 0066a40..c60c338 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
@@ -103,6 +103,11 @@ class CassandraMailboxMapperTest {
CassandraConfiguration.DEFAULT_CONFIGURATION,
cassandraCluster.getCassandraConsistenciesConfiguration());
versionDAO = new CassandraSchemaVersionDAO(cassandra.getConf());
+
+ versionDAO.truncateVersion()
+ .then(versionDAO.updateVersion(new SchemaVersion(7)))
+ .block();
+
testee = new CassandraMailboxMapper(
mailboxDAO,
mailboxPathDAO,
@@ -830,8 +835,6 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInV1Table() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -850,8 +853,6 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInBothTables() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -872,8 +873,6 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldReturnMailboxesWhenExistsInV2Table() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -911,8 +910,6 @@ class CassandraMailboxMapperTest {
@Test
void hasChildrenShouldReturnChildWhenExistsInV1Table() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -932,8 +929,6 @@ class CassandraMailboxMapperTest {
@Test
void hasChildrenShouldReturnChildWhenExistsInBothTables() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathDAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -955,8 +950,6 @@ class CassandraMailboxMapperTest {
@Test
void hasChildrenShouldReturnChildWhenExistsInV2Table() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX)
.block();
mailboxPathV2DAO.save(MAILBOX_PATH, MAILBOX_ID)
@@ -995,8 +988,6 @@ class CassandraMailboxMapperTest {
@Test
void findMailboxWithPathLikeShouldRemoveDuplicatesAndKeepV3() {
- versionDAO.updateVersion(new SchemaVersion(7)).block();
-
mailboxDAO.save(MAILBOX).block();
mailboxPathV3DAO.save(MAILBOX).block();
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 13/22: JAMES-3409 Fix warnings in
MailboxMergingTaskRunner
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 41a2b5858be918b68ef5608c39e3a881b654fb3c
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Fri Jul 31 09:36:49 2020 +0700
JAMES-3409 Fix warnings in MailboxMergingTaskRunner
---
.../james/mailbox/cassandra/mail/task/MailboxMergingTaskRunner.java | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MailboxMergingTaskRunner.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MailboxMergingTaskRunner.java
index 08e55f5..4231d50 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MailboxMergingTaskRunner.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MailboxMergingTaskRunner.java
@@ -28,7 +28,6 @@ import org.apache.james.mailbox.cassandra.ids.CassandraId;
import org.apache.james.mailbox.cassandra.mail.CassandraACLMapper;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
-import org.apache.james.mailbox.cassandra.mail.CassandraUserMailboxRightsDAO;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.model.ComposedMessageId;
import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
@@ -51,17 +50,15 @@ public class MailboxMergingTaskRunner {
private final StoreMessageIdManager messageIdManager;
private final CassandraMessageIdDAO cassandraMessageIdDAO;
private final CassandraMailboxDAO mailboxDAO;
- private final CassandraUserMailboxRightsDAO rightsDAO;
private final CassandraACLMapper cassandraACLMapper;
private final MailboxSession mailboxSession;
@Inject
- public MailboxMergingTaskRunner(MailboxManager mailboxManager, StoreMessageIdManager messageIdManager, CassandraMessageIdDAO cassandraMessageIdDAO, CassandraMailboxDAO mailboxDAO, CassandraUserMailboxRightsDAO rightsDAO, CassandraACLMapper cassandraACLMapper) throws MailboxException {
+ public MailboxMergingTaskRunner(MailboxManager mailboxManager, StoreMessageIdManager messageIdManager, CassandraMessageIdDAO cassandraMessageIdDAO, CassandraMailboxDAO mailboxDAO, CassandraACLMapper cassandraACLMapper) {
this.mailboxSession = mailboxManager.createSystemSession(Username.of("task"));
this.messageIdManager = messageIdManager;
this.cassandraMessageIdDAO = cassandraMessageIdDAO;
this.mailboxDAO = mailboxDAO;
- this.rightsDAO = rightsDAO;
this.cassandraACLMapper = cassandraACLMapper;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 04/22: JAMES-3277 Avoid a double message read upon
CassandraMessageIdMapper::setFlags
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 8b8183153070fedf984954d12f14a30aea16e491
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Mon Oct 12 12:20:27 2020 +0700
JAMES-3277 Avoid a double message read upon CassandraMessageIdMapper::setFlags
Not found was handled via an extra read but we could rely with a smarter error management
on the reads of updateFlags to detect them.
---
.../cassandra/mail/CassandraMessageIdMapper.java | 25 +++++----------------
.../mail/CassandraMessageIdMapperTest.java | 26 ++++++++++++++++++++++
2 files changed, 31 insertions(+), 20 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
index b0293d1..771c276 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapper.java
@@ -101,7 +101,7 @@ public class CassandraMessageIdMapper implements MessageIdMapper {
@Override
public Flux<MailboxMessage> findReactive(Collection<MessageId> messageIds, FetchType fetchType) {
- return Flux.fromStream(messageIds.stream())
+ return Flux.fromIterable(messageIds)
.flatMap(messageId -> imapUidDAO.retrieve((CassandraMessageId) messageId, Optional.empty()), cassandraConfiguration.getMessageReadChunkSize())
.flatMap(composedMessageId -> messageDAO.retrieveMessage(composedMessageId, fetchType)
.map(messageRepresentation -> Pair.of(composedMessageId, messageRepresentation)), cassandraConfiguration.getMessageReadChunkSize())
@@ -220,21 +220,14 @@ public class CassandraMessageIdMapper implements MessageIdMapper {
return Flux.fromIterable(mailboxIds)
.distinct()
.map(mailboxId -> (CassandraId) mailboxId)
- .filterWhen(mailboxId -> haveMetaData(messageId, mailboxId))
.concatMap(mailboxId -> flagsUpdateWithRetry(newState, updateMode, mailboxId, messageId))
.flatMap(this::updateCounts)
.collect(Guavate.toImmutableListMultimap(Pair::getLeft, Pair::getRight))
.block();
}
- private Mono<Boolean> haveMetaData(MessageId messageId, CassandraId mailboxId) {
- return imapUidDAO.retrieve((CassandraMessageId) messageId, Optional.of(mailboxId))
- .hasElements();
- }
-
private Flux<Pair<MailboxId, UpdatedFlags>> flagsUpdateWithRetry(Flags newState, MessageManager.FlagsUpdateMode updateMode, MailboxId mailboxId, MessageId messageId) {
- return Mono.defer(() -> tryFlagsUpdate(newState, updateMode, mailboxId, messageId))
- .single()
+ return Mono.defer(() -> updateFlags(mailboxId, messageId, newState, updateMode))
.retry(cassandraConfiguration.getFlagsUpdateMessageIdMaxRetry())
.onErrorResume(MailboxDeleteDuringUpdateException.class, e -> {
LOGGER.info("Mailbox {} was deleted during flag update", mailboxId);
@@ -261,16 +254,7 @@ public class CassandraMessageIdMapper implements MessageIdMapper {
.thenReturn(pair);
}
- private Mono<List<Pair<Flags, ComposedMessageIdWithMetaData>>> tryFlagsUpdate(Flags newState, MessageManager.FlagsUpdateMode updateMode, MailboxId mailboxId, MessageId messageId) {
- try {
- return updateFlags(mailboxId, messageId, newState, updateMode);
- } catch (MailboxException e) {
- LOGGER.error("Error while updating flags on mailbox: {}", mailboxId);
- return Mono.empty();
- }
- }
-
- private Mono<List<Pair<Flags, ComposedMessageIdWithMetaData>>> updateFlags(MailboxId mailboxId, MessageId messageId, Flags newState, MessageManager.FlagsUpdateMode updateMode) throws MailboxException {
+ private Mono<List<Pair<Flags, ComposedMessageIdWithMetaData>>> updateFlags(MailboxId mailboxId, MessageId messageId, Flags newState, MessageManager.FlagsUpdateMode updateMode) {
CassandraId cassandraId = (CassandraId) mailboxId;
return imapUidDAO.retrieve((CassandraMessageId) messageId, Optional.of(cassandraId))
.flatMap(oldComposedId -> updateFlags(newState, updateMode, cassandraId, oldComposedId))
@@ -300,6 +284,7 @@ public class CassandraMessageIdMapper implements MessageIdMapper {
return imapUidDAO.updateMetadata(newComposedId, oldComposedId.getModSeq())
.filter(FunctionalUtils.identityPredicate())
.flatMap(any -> messageIdDAO.updateMetadata(newComposedId)
- .thenReturn(Pair.of(oldComposedId.getFlags(), newComposedId)));
+ .thenReturn(Pair.of(oldComposedId.getFlags(), newComposedId)))
+ .single();
}
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
index c196c71..fae47aa 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdMapperTest.java
@@ -85,6 +85,32 @@ class CassandraMessageIdMapperTest extends MessageIdMapperTest {
.containsOnly(message1, message2, message3, message4);
}
+ @Test
+ void setFlagsShouldMinimizeMessageReads(CassandraCluster cassandra) throws Exception {
+ CassandraMessageId.Factory messageIdFactory = new CassandraMessageId.Factory();
+ CassandraMailboxSessionMapperFactory mapperFactory = TestCassandraMailboxSessionMapperFactory.forTests(
+ cassandraCluster.getCassandraCluster(),
+ messageIdFactory,
+ CassandraConfiguration.builder()
+ .messageReadChunkSize(3)
+ .build());
+
+ saveMessages();
+
+ StatementRecorder statementRecorder = new StatementRecorder();
+ cassandra.getConf().recordStatements(statementRecorder);
+
+ mapperFactory.getMessageIdMapper(MAILBOX_SESSION).setFlags(message1.getMessageId(),
+ ImmutableList.of(message1.getMailboxId()),
+ new Flags(Flags.Flag.DELETED),
+ MessageManager.FlagsUpdateMode.REPLACE);
+
+ assertThat(statementRecorder.listExecutedStatements(
+ StatementRecorder.Selector.preparedStatementStartingWith("SELECT messageId,mailboxId,uid,modSeq,flagAnswered,flagDeleted," +
+ "flagDraft,flagFlagged,flagRecent,flagSeen,flagUser,userFlags FROM imapUidTable")))
+ .hasSize(1);
+ }
+
@Nested
class FailureTest {
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 19/22: JAMES-3407 Read repair for mailbox entity
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 390f64f5311fa230e371301de815a26b5cb2e679
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Oct 8 16:39:45 2020 +0700
JAMES-3407 Read repair for mailbox entity
Hard coded read repair chance (10%)
---
.../cassandra/mail/CassandraMailboxMapper.java | 75 +++++++++++++++---
.../task/SolveMailboxInconsistenciesService.java | 65 +++++++++-------
.../cassandra/mail/CassandraMailboxMapperTest.java | 91 +++++++++++++++++++++-
3 files changed, 188 insertions(+), 43 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
index 84b5b24..e6d868d 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapper.java
@@ -19,6 +19,7 @@
package org.apache.james.mailbox.cassandra.mail;
+import java.security.SecureRandom;
import java.time.Duration;
import javax.inject.Inject;
@@ -29,6 +30,7 @@ import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.core.Username;
import org.apache.james.mailbox.acl.ACLDiff;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
+import org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesService;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.MailboxExistsException;
import org.apache.james.mailbox.exception.MailboxNotFoundException;
@@ -58,6 +60,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
private static final Duration MIN_RETRY_BACKOFF = Duration.ofMillis(10);
private static final Duration MAX_RETRY_BACKOFF = Duration.ofMillis(1000);
private static final SchemaVersion MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION = new SchemaVersion(8);
+ private static final float READ_REPAIR_CHANCE = 0.1f;
private final CassandraMailboxDAO mailboxDAO;
private final CassandraMailboxPathDAOImpl mailboxPathDAO;
@@ -66,6 +69,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
private final CassandraACLMapper cassandraACLMapper;
private final CassandraUserMailboxRightsDAO userMailboxRightsDAO;
private final CassandraSchemaVersionManager versionManager;
+ private final SecureRandom secureRandom;
@Inject
public CassandraMailboxMapper(CassandraMailboxDAO mailboxDAO,
@@ -82,12 +86,58 @@ public class CassandraMailboxMapper implements MailboxMapper {
this.userMailboxRightsDAO = userMailboxRightsDAO;
this.cassandraACLMapper = aclMapper;
this.versionManager = versionManager;
+ this.secureRandom = new SecureRandom();
}
private Mono<Boolean> needMailboxPathPreviousVersionsSupport() {
return versionManager.isBefore(MAILBOX_PATH_V_3_MIGRATION_PERFORMED_VERSION);
}
+ private Mono<Mailbox> performReadRepair(CassandraId id) {
+ if (shouldReadRepair()) {
+ return mailboxDAO.retrieveMailbox(id)
+ .flatMap(mailboxEntry -> SolveMailboxInconsistenciesService.Inconsistency
+ .detectMailboxDaoInconsistency(mailboxEntry,
+ mailboxPathV3DAO.retrieve(mailboxEntry.generateAssociatedPath()))
+ .flatMap(inconsistency ->
+ inconsistency.fix(new SolveMailboxInconsistenciesService.Context(), mailboxDAO, mailboxPathV3DAO)
+ .then(Mono.just(mailboxEntry))));
+ }
+ return mailboxDAO.retrieveMailbox(id);
+ }
+
+ private Mono<Mailbox> performReadRepair(MailboxPath path) {
+ if (shouldReadRepair()) {
+ return mailboxPathV3DAO.retrieve(path)
+ .flatMap(this::performPathReadRepair);
+ }
+ return mailboxPathV3DAO.retrieve(path);
+
+ }
+
+ private Flux<Mailbox> performReadRepair(Flux<Mailbox> pathEntries) {
+ return pathEntries.flatMap(mailboxPathEntry -> {
+ if (shouldReadRepair()) {
+ return performPathReadRepair(mailboxPathEntry);
+ }
+ return Mono.just(mailboxPathEntry);
+ });
+ }
+
+ private Mono<Mailbox> performPathReadRepair(Mailbox mailboxPathEntry) {
+ return SolveMailboxInconsistenciesService.Inconsistency
+ .detectMailboxPathDaoInconsistency(mailboxPathEntry,
+ mailboxDAO.retrieveMailbox((CassandraId) mailboxPathEntry.getMailboxId()))
+ .flatMap(inconsistency ->
+ inconsistency.fix(new SolveMailboxInconsistenciesService.Context(), mailboxDAO, mailboxPathV3DAO)
+ .then(Mono.just(mailboxPathEntry)));
+ }
+
+ private boolean shouldReadRepair() {
+ return secureRandom.nextFloat() < READ_REPAIR_CHANCE;
+ }
+
+
@Override
public Mono<Void> delete(Mailbox mailbox) {
CassandraId mailboxId = (CassandraId) mailbox.getMailboxId();
@@ -111,7 +161,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
@Override
public Mono<Mailbox> findMailboxByPath(MailboxPath path) {
- return mailboxPathV3DAO.retrieve(path)
+ return performReadRepair(path)
.switchIfEmpty(fromPreviousTable(path))
.flatMap(this::addAcl);
}
@@ -127,9 +177,9 @@ public class CassandraMailboxMapper implements MailboxMapper {
}
@Override
- public Mono<Boolean> pathExists(MailboxPath mailboxName) {
- return mailboxPathV3DAO.retrieve(mailboxName)
- .switchIfEmpty(fromPreviousTable(mailboxName))
+ public Mono<Boolean> pathExists(MailboxPath path) {
+ return performReadRepair(path)
+ .switchIfEmpty(fromPreviousTable(path))
.hasElement();
}
@@ -166,7 +216,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
private Mono<Mailbox> retrieveMailbox(CassandraId mailboxId) {
Mono<MailboxACL> acl = retrieveAcl(mailboxId);
- Mono<Mailbox> simpleMailbox = mailboxDAO.retrieveMailbox(mailboxId);
+ Mono<Mailbox> simpleMailbox = performReadRepair(mailboxId);
return acl.zipWith(simpleMailbox, this::addAcl);
}
@@ -186,7 +236,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
String fixedNamespace = query.getFixedNamespace();
Username fixedUser = query.getFixedUser();
- return listMailboxes(fixedNamespace, fixedUser)
+ return performReadRepair(listMailboxes(fixedNamespace, fixedUser))
.filter(mailbox -> query.isPathMatch(mailbox.generateAssociatedPath()))
.distinct(Mailbox::generateAssociatedPath)
.flatMap(this::addAcl);
@@ -259,7 +309,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
@Override
public Mono<Boolean> hasChildren(Mailbox mailbox, char delimiter) {
- return listMailboxes(mailbox.getNamespace(), mailbox.getUser())
+ return performReadRepair(listMailboxes(mailbox.getNamespace(), mailbox.getUser()))
.filter(idAndPath -> isPathChildOfMailbox(idAndPath, mailbox, delimiter))
.hasElements();
}
@@ -270,7 +320,7 @@ public class CassandraMailboxMapper implements MailboxMapper {
@Override
public Flux<Mailbox> list() {
- return mailboxDAO.retrieveAllMailboxes()
+ return performReadRepair(mailboxDAO.retrieveAllMailboxes())
.flatMap(this::toMailboxWithAcl);
}
@@ -307,9 +357,10 @@ public class CassandraMailboxMapper implements MailboxMapper {
@Override
public Flux<Mailbox> findNonPersonalMailboxes(Username userName, Right right) {
- return userMailboxRightsDAO.listRightsForUser(userName)
- .filter(mailboxId -> mailboxId.getRight().contains(right))
- .map(Pair::getLeft)
- .flatMap(this::retrieveMailbox);
+ return performReadRepair(
+ userMailboxRightsDAO.listRightsForUser(userName)
+ .filter(mailboxId -> mailboxId.getRight().contains(right))
+ .map(Pair::getLeft)
+ .flatMap(this::retrieveMailbox));
}
}
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
index 8aacb18..8266ceb 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMailboxInconsistenciesService.java
@@ -50,7 +50,36 @@ public class SolveMailboxInconsistenciesService {
public static final Logger LOGGER = LoggerFactory.getLogger(SolveMailboxInconsistenciesService.class);
@FunctionalInterface
- interface Inconsistency {
+ public interface Inconsistency {
+ static Mono<Inconsistency> detectMailboxDaoInconsistency(Mailbox mailboxEntry, Mono<Mailbox> pathEntry) {
+ return pathEntry
+ .map(mailboxByPath -> {
+ if (mailboxByPath.getMailboxId().equals(mailboxEntry.getMailboxId())) {
+ return NO_INCONSISTENCY;
+ }
+ // Path entry references another mailbox.
+ return new ConflictingEntryInconsistency(ConflictingEntry.builder()
+ .mailboxDaoEntry(mailboxEntry)
+ .mailboxPathDaoEntry(mailboxByPath));
+ })
+ .defaultIfEmpty(new OrphanMailboxDAOEntry(mailboxEntry));
+ }
+
+
+ static Mono<Inconsistency> detectMailboxPathDaoInconsistency(Mailbox mailboxByPathEntry, Mono<Mailbox> mailboxEntry) {
+ return mailboxEntry
+ .map(mailboxById -> {
+ if (mailboxByPathEntry.generateAssociatedPath().equals(mailboxById.generateAssociatedPath())) {
+ return NO_INCONSISTENCY;
+ }
+ // Mailbox references another path
+ return new ConflictingEntryInconsistency(ConflictingEntry.builder()
+ .mailboxDaoEntry(mailboxById)
+ .mailboxPathDaoEntry(mailboxByPathEntry));
+ })
+ .defaultIfEmpty(new OrphanMailboxPathDAOEntry(mailboxByPathEntry));
+ }
+
Mono<Result> fix(Context context, CassandraMailboxDAO mailboxDAO, CassandraMailboxPathV3DAO pathV3DAO);
}
@@ -295,7 +324,7 @@ public class SolveMailboxInconsistenciesService {
private final ConcurrentLinkedDeque<ConflictingEntry> conflictingEntries;
private final AtomicLong errors;
- Context() {
+ public Context() {
this(new AtomicLong(), new AtomicLong(), ImmutableList.of(), ImmutableList.of(), new AtomicLong());
}
@@ -391,33 +420,15 @@ public class SolveMailboxInconsistenciesService {
.doOnNext(any -> context.incrementProcessedMailboxEntries());
}
- private Mono<Inconsistency> detectMailboxDaoInconsistency(Mailbox mailbox) {
- return mailboxPathV3DAO.retrieve(mailbox.generateAssociatedPath())
- .map(mailboxByPath -> {
- if (mailboxByPath.getMailboxId().equals(mailbox.getMailboxId())) {
- return NO_INCONSISTENCY;
- }
- // Path entry references another mailbox.
- return new ConflictingEntryInconsistency(ConflictingEntry.builder()
- .mailboxDaoEntry(mailbox)
- .mailboxPathDaoEntry(mailboxByPath));
- })
- .defaultIfEmpty(new OrphanMailboxDAOEntry(mailbox));
+ private Mono<Inconsistency> detectMailboxDaoInconsistency(Mailbox mailboxEntry) {
+ Mono<Mailbox> pathEntry = mailboxPathV3DAO.retrieve(mailboxEntry.generateAssociatedPath());
+ return Inconsistency.detectMailboxDaoInconsistency(mailboxEntry, pathEntry);
}
- private Mono<Inconsistency> detectMailboxPathDaoInconsistency(Mailbox mailboxByPath) {
- CassandraId cassandraId = (CassandraId) mailboxByPath.getMailboxId();
+ private Mono<Inconsistency> detectMailboxPathDaoInconsistency(Mailbox mailboxByPathEntry) {
+ CassandraId cassandraId = (CassandraId) mailboxByPathEntry.getMailboxId();
- return mailboxDAO.retrieveMailbox(cassandraId)
- .map(mailboxById -> {
- if (mailboxByPath.generateAssociatedPath().equals(mailboxById.generateAssociatedPath())) {
- return NO_INCONSISTENCY;
- }
- // Mailbox references another path
- return new ConflictingEntryInconsistency(ConflictingEntry.builder()
- .mailboxDaoEntry(mailboxById)
- .mailboxPathDaoEntry(mailboxByPath));
- })
- .defaultIfEmpty(new OrphanMailboxPathDAOEntry(mailboxByPath));
+ Mono<Mailbox> mailboxEntry = mailboxDAO.retrieveMailbox(cassandraId);
+ return Inconsistency.detectMailboxPathDaoInconsistency(mailboxByPathEntry, mailboxEntry);
}
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
index c60c338..0be1024 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxMapperTest.java
@@ -25,6 +25,7 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.util.List;
+import java.util.stream.IntStream;
import org.apache.commons.lang3.StringUtils;
import org.apache.james.backends.cassandra.CassandraCluster;
@@ -50,6 +51,7 @@ import org.apache.james.mailbox.model.UidValidity;
import org.apache.james.mailbox.model.search.ExactName;
import org.apache.james.mailbox.model.search.MailboxQuery;
import org.apache.james.mailbox.model.search.Wildcard;
+import org.apache.james.mailbox.store.MailboxReactorUtils;
import org.assertj.core.api.SoftAssertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Disabled;
@@ -60,6 +62,8 @@ import org.junit.jupiter.api.extension.RegisterExtension;
import com.github.fge.lambdas.Throwing;
import com.github.fge.lambdas.runnable.ThrowingRunnable;
+import reactor.core.publisher.Mono;
+
class CassandraMailboxMapperTest {
private static final UidValidity UID_VALIDITY = UidValidity.of(52);
private static final Username USER = Username.of("user");
@@ -186,8 +190,8 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
- .times(1)
- .whenQueryStartsWith("INSERT INTO mailbox (id,name,uidvalidity,mailboxbase) VALUES (:id,:name,:uidvalidity,:mailboxbase);"));
+ .times(1)
+ .whenQueryStartsWith("INSERT INTO mailbox (id,name,uidvalidity,mailboxbase) VALUES (:id,:name,:uidvalidity,:mailboxbase);"));
testee.rename(inboxRenamed).block();
@@ -236,8 +240,8 @@ class CassandraMailboxMapperTest {
cassandra.getConf()
.registerScenario(fail()
- .times(1)
- .whenQueryStartsWith("DELETE FROM mailbox WHERE id=:id;"));
+ .times(1)
+ .whenQueryStartsWith("DELETE FROM mailbox WHERE id=:id;"));
testee.delete(inbox).block();
@@ -253,6 +257,85 @@ class CassandraMailboxMapperTest {
}
}
+ @Nested
+ class ReadRepairs {
+ @Test
+ void findMailboxByIdShouldEventuallyFixInconsistencyWhenMailboxIsNotInPath() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+
+ IntStream.range(0, 100).forEach(i ->
+ testee.findMailboxById(MAILBOX_ID)
+ .onErrorResume(e -> Mono.empty())
+ .block());
+
+ SoftAssertions.assertSoftly(Throwing.consumer(softly -> {
+ softly(softly)
+ .assertThat(testee.findMailboxById(MAILBOX_ID).block())
+ .isEqualTo(MAILBOX);
+ softly(softly)
+ .assertThat(testee.findMailboxByPath(MAILBOX_PATH).block())
+ .isEqualTo(MAILBOX);
+ }));
+ }
+
+ @Test
+ void orphanMailboxIdEntriesCanNotBeReadRepaired() {
+ mailboxDAO.save(MAILBOX)
+ .block();
+
+ IntStream.range(0, 100).forEach(i ->
+ testee.findMailboxByPath(MAILBOX_PATH)
+ .onErrorResume(e -> Mono.empty())
+ .block());
+
+ SoftAssertions.assertSoftly(Throwing.consumer(softly -> {
+ softly(softly)
+ .assertThat(testee.findMailboxById(MAILBOX_ID).block())
+ .isEqualTo(MAILBOX);
+ softly.assertThat(MailboxReactorUtils.blockOptional(testee.findMailboxByPath(MAILBOX_PATH)))
+ .isEmpty();
+ }));
+ }
+
+ @Test
+ void orphanPathEntriesCanNotBeRepairedByIdReads() {
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ IntStream.range(0, 100).forEach(i ->
+ testee.findMailboxById(MAILBOX_ID)
+ .onErrorResume(e -> Mono.empty())
+ .block());
+
+ SoftAssertions.assertSoftly(Throwing.consumer(softly -> {
+ softly.assertThatThrownBy(() -> MailboxReactorUtils.blockOptional(testee.findMailboxById(MAILBOX_ID)))
+ .isInstanceOf(MailboxNotFoundException.class);
+ softly(softly)
+ .assertThat(testee.findMailboxByPath(MAILBOX_PATH).block())
+ .isEqualTo(MAILBOX);
+ }));
+ }
+
+ @Test
+ void findMailboxByPathShouldFixInconsistencyWhenMailboxIsNotReferencedById() {
+ mailboxPathV3DAO.save(MAILBOX)
+ .block();
+
+ IntStream.range(0, 100).forEach(i ->
+ testee.findMailboxByPath(MAILBOX_PATH)
+ .onErrorResume(e -> Mono.empty())
+ .block());
+
+ SoftAssertions.assertSoftly(Throwing.consumer(softly -> {
+ softly.assertThatThrownBy(() -> MailboxReactorUtils.blockOptional(testee.findMailboxById(MAILBOX_ID)))
+ .isInstanceOf(MailboxNotFoundException.class);
+ softly.assertThat(MailboxReactorUtils.blockOptional(testee.findMailboxByPath(MAILBOX_PATH)))
+ .isEmpty();
+ }));
+ }
+ }
+
@Disabled("In order to be more performant mailboxPath V3 table includes the UID_VALIDITY." +
"Reading paths no longer requires reading the mailbox by id but this of course has a " +
"consistency cost.")
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 12/22: JAMES-3409 Singleton binding for
CassandraMailboxPathV3DAO
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f1bbf77a8d2028822d2209434edff10dbf140a5f
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Jul 30 09:24:34 2020 +0700
JAMES-3409 Singleton binding for CassandraMailboxPathV3DAO
---
.../java/org/apache/james/modules/mailbox/CassandraMailboxModule.java | 2 ++
1 file changed, 2 insertions(+)
diff --git a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java
index a3a0742..fc8c038 100644
--- a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java
+++ b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/mailbox/CassandraMailboxModule.java
@@ -53,6 +53,7 @@ import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxMapper;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathDAOImpl;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMailboxRecentsDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
@@ -121,6 +122,7 @@ public class CassandraMailboxModule extends AbstractModule {
bind(CassandraMailboxDAO.class).in(Scopes.SINGLETON);
bind(CassandraMailboxPathDAOImpl.class).in(Scopes.SINGLETON);
bind(CassandraMailboxPathV2DAO.class).in(Scopes.SINGLETON);
+ bind(CassandraMailboxPathV3DAO.class).in(Scopes.SINGLETON);
bind(CassandraMailboxRecentsDAO.class).in(Scopes.SINGLETON);
bind(CassandraMessageDAO.class).in(Scopes.SINGLETON);
bind(CassandraMessageIdDAO.class).in(Scopes.SINGLETON);
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 09/22: JAMES-3409 CassandraMailboxPathV3DAO (with
UidValidity)
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 33da147943ae0a5631841fea7e75ab242251b6c6
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Jul 29 16:48:50 2020 +0700
JAMES-3409 CassandraMailboxPathV3DAO (with UidValidity)
---
.../cassandra/mail/CassandraMailboxPathV3DAO.java | 215 +++++++++++++++++++++
.../cassandra/modules/CassandraMailboxModule.java | 13 ++
.../table/CassandraMailboxPathV3Table.java} | 34 ++--
.../mail/CassandraMailboxPathV3DAOTest.java | 153 +++++++++++++++
.../mailbox/cassandra/mail/MailboxFixture.java | 6 +
5 files changed, 404 insertions(+), 17 deletions(-)
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAO.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAO.java
new file mode 100644
index 0000000..c07c702
--- /dev/null
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAO.java
@@ -0,0 +1,215 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail;
+
+import static com.datastax.driver.core.querybuilder.QueryBuilder.bindMarker;
+import static com.datastax.driver.core.querybuilder.QueryBuilder.eq;
+import static com.datastax.driver.core.querybuilder.QueryBuilder.insertInto;
+import static com.datastax.driver.core.querybuilder.QueryBuilder.select;
+import static org.apache.james.mailbox.cassandra.GhostMailbox.TYPE;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.FIELDS;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.MAILBOX_ID;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.MAILBOX_NAME;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.NAMESPACE;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.TABLE_NAME;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.UIDVALIDITY;
+import static org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table.USER;
+
+import javax.inject.Inject;
+
+import org.apache.james.backends.cassandra.init.configuration.CassandraConsistenciesConfiguration;
+import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor;
+import org.apache.james.backends.cassandra.utils.CassandraUtils;
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.cassandra.GhostMailbox;
+import org.apache.james.mailbox.cassandra.ids.CassandraId;
+import org.apache.james.mailbox.model.Mailbox;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UidValidity;
+import org.apache.james.util.FunctionalUtils;
+import org.apache.james.util.ReactorUtils;
+
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.PreparedStatement;
+import com.datastax.driver.core.Row;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+
+import reactor.core.publisher.Flux;
+import reactor.core.publisher.Mono;
+
+public class CassandraMailboxPathV3DAO {
+ private final CassandraAsyncExecutor cassandraAsyncExecutor;
+ private final CassandraUtils cassandraUtils;
+ private final PreparedStatement delete;
+ private final PreparedStatement insert;
+ private final PreparedStatement select;
+ private final PreparedStatement selectUser;
+ private final PreparedStatement selectAll;
+ private final ConsistencyLevel consistencyLevel;
+
+ @Inject
+ public CassandraMailboxPathV3DAO(Session session, CassandraUtils cassandraUtils,
+ CassandraConsistenciesConfiguration consistenciesConfiguration) {
+ this.cassandraAsyncExecutor = new CassandraAsyncExecutor(session);
+ this.consistencyLevel = consistenciesConfiguration.getLightweightTransaction();
+ this.cassandraUtils = cassandraUtils;
+ this.insert = prepareInsert(session);
+ this.delete = prepareDelete(session);
+ this.select = prepareSelect(session);
+ this.selectUser = prepareSelectUser(session);
+ this.selectAll = prepareSelectAll(session);
+ }
+
+ private PreparedStatement prepareDelete(Session session) {
+ return session.prepare(QueryBuilder.delete()
+ .from(TABLE_NAME)
+ .where(eq(NAMESPACE, bindMarker(NAMESPACE)))
+ .and(eq(USER, bindMarker(USER)))
+ .and(eq(MAILBOX_NAME, bindMarker(MAILBOX_NAME)))
+ .ifExists());
+ }
+
+ private PreparedStatement prepareInsert(Session session) {
+ return session.prepare(insertInto(TABLE_NAME)
+ .value(NAMESPACE, bindMarker(NAMESPACE))
+ .value(USER, bindMarker(USER))
+ .value(MAILBOX_NAME, bindMarker(MAILBOX_NAME))
+ .value(MAILBOX_ID, bindMarker(MAILBOX_ID))
+ .value(UIDVALIDITY, bindMarker(UIDVALIDITY))
+ .ifNotExists());
+ }
+
+ private PreparedStatement prepareSelect(Session session) {
+ return session.prepare(select(FIELDS)
+ .from(TABLE_NAME)
+ .where(eq(NAMESPACE, bindMarker(NAMESPACE)))
+ .and(eq(USER, bindMarker(USER)))
+ .and(eq(MAILBOX_NAME, bindMarker(MAILBOX_NAME))));
+ }
+
+ private PreparedStatement prepareSelectUser(Session session) {
+ return session.prepare(select(FIELDS)
+ .from(TABLE_NAME)
+ .where(eq(NAMESPACE, bindMarker(NAMESPACE)))
+ .and(eq(USER, bindMarker(USER))));
+ }
+
+ private PreparedStatement prepareSelectAll(Session session) {
+ return session.prepare(select(FIELDS)
+ .from(TABLE_NAME));
+ }
+
+ public Mono<Mailbox> retrieve(MailboxPath mailboxPath) {
+ return cassandraAsyncExecutor.executeSingleRow(
+ select.bind()
+ .setString(NAMESPACE, mailboxPath.getNamespace())
+ .setString(USER, sanitizeUser(mailboxPath.getUser()))
+ .setString(MAILBOX_NAME, mailboxPath.getName())
+ .setConsistencyLevel(consistencyLevel))
+ .map(this::fromRowToCassandraIdAndPath)
+ .map(FunctionalUtils.toFunction(this::logGhostMailboxSuccess))
+ .switchIfEmpty(ReactorUtils.executeAndEmpty(() -> logGhostMailboxFailure(mailboxPath)));
+ }
+
+ public Flux<Mailbox> listUserMailboxes(String namespace, Username user) {
+ return cassandraAsyncExecutor.execute(
+ selectUser.bind()
+ .setString(NAMESPACE, namespace)
+ .setString(USER, sanitizeUser(user))
+ .setConsistencyLevel(consistencyLevel))
+ .flatMapMany(cassandraUtils::convertToFlux)
+ .map(this::fromRowToCassandraIdAndPath)
+ .map(FunctionalUtils.toFunction(this::logReadSuccess));
+ }
+
+ public Flux<Mailbox> listAll() {
+ return cassandraAsyncExecutor.execute(
+ selectAll.bind())
+ .flatMapMany(cassandraUtils::convertToFlux)
+ .map(this::fromRowToCassandraIdAndPath)
+ .map(FunctionalUtils.toFunction(this::logReadSuccess));
+ }
+
+ /**
+ * See https://issues.apache.org/jira/browse/MAILBOX-322 to read about the Ghost mailbox bug.
+ *
+ * A missed read on an existing mailbox is the cause of the ghost mailbox bug. Here we log missing reads. Successful
+ * reads and write operations are also added in order to allow audit in order to know if the mailbox existed.
+ */
+ public void logGhostMailboxSuccess(Mailbox value) {
+ logReadSuccess(value);
+ }
+
+ public void logGhostMailboxFailure(MailboxPath mailboxPath) {
+ GhostMailbox.logger()
+ .addField(GhostMailbox.MAILBOX_NAME, mailboxPath)
+ .addField(TYPE, "readMiss")
+ .log(logger -> logger.debug("Read mailbox missed"));
+ }
+
+ /**
+ * See https://issues.apache.org/jira/browse/MAILBOX-322 to read about the Ghost mailbox bug.
+ *
+ * Read success allows to know if a mailbox existed before (mailbox write history might be older than this log introduction
+ * or log history might have been dropped)
+ */
+ private void logReadSuccess(Mailbox mailbox) {
+ GhostMailbox.logger()
+ .addField(GhostMailbox.MAILBOX_NAME, mailbox.generateAssociatedPath())
+ .addField(TYPE, "readSuccess")
+ .addField(GhostMailbox.MAILBOX_ID, mailbox.getMailboxId())
+ .log(logger -> logger.debug("Read mailbox succeeded"));
+ }
+
+ private Mailbox fromRowToCassandraIdAndPath(Row row) {
+ return new Mailbox(
+ new MailboxPath(row.getString(NAMESPACE),
+ Username.of(row.getString(USER)),
+ row.getString(MAILBOX_NAME)),
+ UidValidity.of(row.getLong(UIDVALIDITY)),
+ CassandraId.of(row.getUUID(MAILBOX_ID)));
+ }
+
+ public Mono<Boolean> save(Mailbox mailbox) {
+ CassandraId id = (CassandraId) mailbox.getMailboxId();
+
+ return cassandraAsyncExecutor.executeReturnApplied(insert.bind()
+ .setString(NAMESPACE, mailbox.getNamespace())
+ .setString(USER, sanitizeUser(mailbox.getUser()))
+ .setLong(UIDVALIDITY, mailbox.getUidValidity().asLong())
+ .setString(MAILBOX_NAME, mailbox.getName())
+ .setUUID(MAILBOX_ID, id.asUuid()));
+ }
+
+ public Mono<Void> delete(MailboxPath mailboxPath) {
+ return cassandraAsyncExecutor.executeVoid(delete.bind()
+ .setString(NAMESPACE, mailboxPath.getNamespace())
+ .setString(USER, sanitizeUser(mailboxPath.getUser()))
+ .setString(MAILBOX_NAME, mailboxPath.getName()));
+ }
+
+ private String sanitizeUser(Username user) {
+ if (user == null) {
+ return "";
+ }
+ return user.asString();
+ }
+}
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/modules/CassandraMailboxModule.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/modules/CassandraMailboxModule.java
index d4341aa..62e9fb5 100644
--- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/modules/CassandraMailboxModule.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/modules/CassandraMailboxModule.java
@@ -27,6 +27,7 @@ import org.apache.james.backends.cassandra.components.CassandraModule;
import org.apache.james.backends.cassandra.utils.CassandraConstants;
import org.apache.james.mailbox.cassandra.table.CassandraMailboxPathTable;
import org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV2Table;
+import org.apache.james.mailbox.cassandra.table.CassandraMailboxPathV3Table;
import org.apache.james.mailbox.cassandra.table.CassandraMailboxTable;
import com.datastax.driver.core.schemabuilder.SchemaBuilder;
@@ -68,5 +69,17 @@ public interface CassandraMailboxModule {
.addPartitionKey(CassandraMailboxPathV2Table.USER, text())
.addClusteringColumn(CassandraMailboxPathV2Table.MAILBOX_NAME, text())
.addColumn(CassandraMailboxPathV2Table.MAILBOX_ID, timeuuid()))
+ .table(CassandraMailboxPathV3Table.TABLE_NAME)
+ .comment("Denormalisation table. Allow to retrieve mailboxes belonging to a certain user. This is a " +
+ "LIST optimisation.")
+ .options(options -> options
+ .caching(SchemaBuilder.KeyCaching.ALL,
+ SchemaBuilder.rows(CassandraConstants.DEFAULT_CACHED_ROW_PER_PARTITION)))
+ .statement(statement -> statement
+ .addPartitionKey(CassandraMailboxPathV3Table.NAMESPACE, text())
+ .addPartitionKey(CassandraMailboxPathV3Table.USER, text())
+ .addClusteringColumn(CassandraMailboxPathV3Table.MAILBOX_NAME, text())
+ .addColumn(CassandraMailboxPathV3Table.MAILBOX_ID, timeuuid())
+ .addColumn(CassandraMailboxPathV3Table.UIDVALIDITY, bigint()))
.build();
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/table/CassandraMailboxPathV3Table.java
similarity index 59%
copy from mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java
copy to mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/table/CassandraMailboxPathV3Table.java
index b9f29e8..078824f 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/table/CassandraMailboxPathV3Table.java
@@ -17,21 +17,21 @@
* under the License. *
****************************************************************/
-package org.apache.james.mailbox.cassandra.mail;
-
-import org.apache.james.core.Username;
-import org.apache.james.mailbox.cassandra.ids.CassandraId;
-import org.apache.james.mailbox.model.MailboxPath;
-
-public interface MailboxFixture {
- Username USER = Username.of("user");
- Username OTHER_USER = Username.of("other");
-
- CassandraId INBOX_ID = CassandraId.timeBased();
- CassandraId OUTBOX_ID = CassandraId.timeBased();
- CassandraId otherMailboxId = CassandraId.timeBased();
- MailboxPath USER_INBOX_MAILBOXPATH = MailboxPath.forUser(USER, "INBOX");
- MailboxPath USER_OUTBOX_MAILBOXPATH = MailboxPath.forUser(USER, "OUTBOX");
- MailboxPath OTHER_USER_MAILBOXPATH = MailboxPath.forUser(OTHER_USER, "INBOX");
- CassandraIdAndPath INBOX_ID_AND_PATH = new CassandraIdAndPath(INBOX_ID, USER_INBOX_MAILBOXPATH);
+package org.apache.james.mailbox.cassandra.table;
+
+public interface CassandraMailboxPathV3Table {
+
+ String TABLE_NAME = "mailboxPathV3";
+
+ String NAMESPACE = "namespace";
+
+ String USER = "user";
+
+ String MAILBOX_NAME = "mailboxName";
+
+ String MAILBOX_ID = "mailboxId";
+ String UIDVALIDITY = "uidvalidity";
+
+ String[] FIELDS = { NAMESPACE, USER, MAILBOX_NAME, MAILBOX_ID, UIDVALIDITY};
+
}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAOTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAOTest.java
new file mode 100644
index 0000000..92d1aa9
--- /dev/null
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMailboxPathV3DAOTest.java
@@ -0,0 +1,153 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail;
+
+import static org.apache.james.mailbox.cassandra.mail.MailboxFixture.MAILBOX_1;
+import static org.apache.james.mailbox.cassandra.mail.MailboxFixture.MAILBOX_2;
+import static org.apache.james.mailbox.cassandra.mail.MailboxFixture.MAILBOX_3;
+import static org.apache.james.mailbox.cassandra.mail.MailboxFixture.USER_INBOX_MAILBOXPATH;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+
+import org.apache.james.backends.cassandra.CassandraCluster;
+import org.apache.james.backends.cassandra.CassandraClusterExtension;
+import org.apache.james.backends.cassandra.components.CassandraModule;
+import org.apache.james.backends.cassandra.utils.CassandraUtils;
+import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule;
+import org.apache.james.mailbox.cassandra.modules.CassandraMailboxModule;
+import org.apache.james.mailbox.model.Mailbox;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import nl.jqno.equalsverifier.EqualsVerifier;
+
+class CassandraMailboxPathV3DAOTest {
+ @RegisterExtension
+ static CassandraClusterExtension cassandraCluster = new CassandraClusterExtension(CassandraModule.aggregateModules(
+ CassandraMailboxModule.MODULE, CassandraSchemaVersionModule.MODULE));
+
+ CassandraMailboxPathV3DAO testee;
+
+ @BeforeEach
+ void setUp(CassandraCluster cassandra) {
+ testee = new CassandraMailboxPathV3DAO(
+ cassandra.getConf(),
+ CassandraUtils.WITH_DEFAULT_CONFIGURATION,
+ cassandraCluster.getCassandraConsistenciesConfiguration());
+ }
+
+ @Test
+ void cassandraIdAndPathShouldRespectBeanContract() {
+ EqualsVerifier.forClass(CassandraIdAndPath.class).verify();
+ }
+
+ @Test
+ void saveShouldInsertNewEntry() {
+ assertThat(testee.save(MAILBOX_1).block()).isTrue();
+
+ assertThat(testee.retrieve(USER_INBOX_MAILBOXPATH).blockOptional())
+ .contains(MAILBOX_1);
+ }
+
+ @Test
+ void saveOnSecondShouldBeFalse() {
+ assertThat(testee.save(MAILBOX_1).block()).isTrue();
+ assertThat(testee.save(MAILBOX_1).block()).isFalse();
+ }
+
+ @Test
+ void retrieveIdShouldReturnEmptyWhenEmptyData() {
+ assertThat(testee.retrieve(USER_INBOX_MAILBOXPATH).blockOptional())
+ .isEmpty();
+ }
+
+ @Test
+ void retrieveIdShouldReturnStoredData() {
+ testee.save(MAILBOX_1).block();
+
+ assertThat(testee.retrieve(USER_INBOX_MAILBOXPATH).blockOptional())
+ .contains(MAILBOX_1);
+ }
+
+ @Test
+ void getUserMailboxesShouldReturnAllMailboxesOfUser() {
+ testee.save(MAILBOX_1).block();
+ testee.save(MAILBOX_2).block();
+ testee.save(MAILBOX_3).block();
+
+ List<Mailbox> cassandraIds = testee
+ .listUserMailboxes(USER_INBOX_MAILBOXPATH.getNamespace(), USER_INBOX_MAILBOXPATH.getUser())
+ .collectList()
+ .block();
+
+ assertThat(cassandraIds)
+ .hasSize(2)
+ .containsOnly(MAILBOX_1, MAILBOX_2);
+ }
+
+ @Test
+ void deleteShouldNotThrowWhenEmpty() {
+ testee.delete(USER_INBOX_MAILBOXPATH).block();
+ }
+
+ @Test
+ void deleteShouldDeleteTheExistingMailboxId() {
+ testee.save(MAILBOX_1).block();
+
+ testee.delete(USER_INBOX_MAILBOXPATH).block();
+
+ assertThat(testee.retrieve(USER_INBOX_MAILBOXPATH).blockOptional())
+ .isEmpty();
+ }
+
+ @Test
+ void listAllShouldBeEmptyByDefault() {
+ assertThat(testee.listAll().collectList().block()).isEmpty();
+ }
+
+ @Test
+ void listAllShouldContainAddedEntry() {
+ testee.save(MAILBOX_1).block();
+
+ assertThat(testee.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX_1);
+ }
+
+ @Test
+ void listAllShouldNotContainDeletedEntry() {
+ testee.save(MAILBOX_1).block();
+
+ testee.delete(USER_INBOX_MAILBOXPATH).block();
+
+ assertThat(testee.listAll().collectList().block())
+ .isEmpty();
+ }
+
+ @Test
+ void listAllShouldContainAddedEntries() {
+ testee.save(MAILBOX_1).block();
+ testee.save(MAILBOX_3).block();
+
+ assertThat(testee.listAll().collectList().block())
+ .containsExactlyInAnyOrder(MAILBOX_1, MAILBOX_3);
+ }
+}
\ No newline at end of file
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java
index b9f29e8..6f2a79b 100644
--- a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/MailboxFixture.java
@@ -21,7 +21,9 @@ package org.apache.james.mailbox.cassandra.mail;
import org.apache.james.core.Username;
import org.apache.james.mailbox.cassandra.ids.CassandraId;
+import org.apache.james.mailbox.model.Mailbox;
import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UidValidity;
public interface MailboxFixture {
Username USER = Username.of("user");
@@ -34,4 +36,8 @@ public interface MailboxFixture {
MailboxPath USER_OUTBOX_MAILBOXPATH = MailboxPath.forUser(USER, "OUTBOX");
MailboxPath OTHER_USER_MAILBOXPATH = MailboxPath.forUser(OTHER_USER, "INBOX");
CassandraIdAndPath INBOX_ID_AND_PATH = new CassandraIdAndPath(INBOX_ID, USER_INBOX_MAILBOXPATH);
+
+ Mailbox MAILBOX_1 = new Mailbox(USER_INBOX_MAILBOXPATH, UidValidity.of(42), INBOX_ID);
+ Mailbox MAILBOX_2 = new Mailbox(USER_OUTBOX_MAILBOXPATH, UidValidity.of(43), OUTBOX_ID);
+ Mailbox MAILBOX_3 = new Mailbox(OTHER_USER_MAILBOXPATH, UidValidity.of(44), otherMailboxId);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 11/22: JAMES-3409 Migration for MailboxPath V3
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 8a3dad0162a118acb3e8c68ffdf6c0edb626a07a
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Wed Jul 29 17:45:28 2020 +0700
JAMES-3409 Migration for MailboxPath V3
---
.../versions/CassandraSchemaVersionManager.java | 2 +-
.../mail/migration/MailboxPathV3Migration.java | 139 +++++++++++++++++++++
...athV3MigrationTaskAdditionalInformationDTO.java | 89 +++++++++++++
.../migration/MailboxPathV3MigrationTaskDTO.java | 59 +++++++++
...ailboxPathV3MigrationTaskSerializationTest.java | 52 ++++++++
.../mail/migration/MailboxPathV3MigrationTest.java | 100 +++++++++++++++
.../modules/webadmin/CassandraRoutesModule.java | 3 +
7 files changed, 443 insertions(+), 1 deletion(-)
diff --git a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/versions/CassandraSchemaVersionManager.java b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/versions/CassandraSchemaVersionManager.java
index 516fbd3..e38037a 100644
--- a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/versions/CassandraSchemaVersionManager.java
+++ b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/versions/CassandraSchemaVersionManager.java
@@ -36,7 +36,7 @@ import reactor.core.publisher.Mono;
public class CassandraSchemaVersionManager {
public static final SchemaVersion MIN_VERSION = new SchemaVersion(5);
- public static final SchemaVersion MAX_VERSION = new SchemaVersion(7);
+ public static final SchemaVersion MAX_VERSION = new SchemaVersion(8);
public static final SchemaVersion DEFAULT_VERSION = MIN_VERSION;
private static final Logger LOGGER = LoggerFactory.getLogger(CassandraSchemaVersionManager.class);
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3Migration.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3Migration.java
new file mode 100644
index 0000000..910c61f
--- /dev/null
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3Migration.java
@@ -0,0 +1,139 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import java.time.Clock;
+import java.time.Instant;
+import java.util.Optional;
+
+import javax.inject.Inject;
+
+import org.apache.james.backends.cassandra.migration.Migration;
+import org.apache.james.mailbox.cassandra.mail.CassandraIdAndPath;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import reactor.core.publisher.Mono;
+
+public class MailboxPathV3Migration implements Migration {
+
+ static class MailboxPathV3MigrationTask implements Task {
+ private final MailboxPathV3Migration migration;
+
+ MailboxPathV3MigrationTask(MailboxPathV3Migration migration) {
+ this.migration = migration;
+ }
+
+ @Override
+ public Result run() throws InterruptedException {
+ return migration.runTask();
+ }
+
+ @Override
+ public TaskType type() {
+ return TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.of(migration.getAdditionalInformation());
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+ private final long remainingCount;
+ private final long initialCount;
+ private final Instant timestamp;
+
+ public AdditionalInformation(long remainingCount, long initialCount, Instant timestamp) {
+ this.remainingCount = remainingCount;
+ this.initialCount = initialCount;
+ this.timestamp = timestamp;
+ }
+
+ public long getRemainingCount() {
+ return remainingCount;
+ }
+
+ public long getInitialCount() {
+ return initialCount;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static final Logger LOGGER = LoggerFactory.getLogger(MailboxPathV3Migration.class);
+ public static final TaskType TYPE = TaskType.of("cassandra-mailbox-path-v3-migration");
+ private final CassandraMailboxPathV2DAO daoV2;
+ private final CassandraMailboxPathV3DAO daoV3;
+ private final CassandraMailboxDAO mailboxDAO;
+ private final long initialCount;
+
+ @Inject
+ public MailboxPathV3Migration(CassandraMailboxPathV2DAO daoV2, CassandraMailboxPathV3DAO daoV3, CassandraMailboxDAO mailboxDAO) {
+ this.daoV2 = daoV2;
+ this.daoV3 = daoV3;
+ this.mailboxDAO = mailboxDAO;
+ this.initialCount = getCurrentCount();
+ }
+
+ @Override
+ public void apply() {
+ daoV2.listAll()
+ .flatMap(this::migrate)
+ .doOnError(t -> LOGGER.error("Error while performing migration", t))
+ .blockLast();
+ }
+
+ private Mono<Void> migrate(CassandraIdAndPath idAndPath) {
+ return mailboxDAO.retrieveMailbox(idAndPath.getCassandraId())
+ .flatMap(mailbox -> daoV3.save(mailbox)
+ .then(daoV2.delete(mailbox.generateAssociatedPath())))
+ .onErrorResume(error -> handleErrorMigrate(idAndPath, error))
+ .then();
+ }
+
+ private Mono<Void> handleErrorMigrate(CassandraIdAndPath idAndPath, Throwable throwable) {
+ LOGGER.error("Error while performing migration for path {}", idAndPath.getMailboxPath(), throwable);
+ return Mono.empty();
+ }
+
+ @Override
+ public Task asTask() {
+ return new MailboxPathV3MigrationTask(this);
+ }
+
+ AdditionalInformation getAdditionalInformation() {
+ return new AdditionalInformation(getCurrentCount(), initialCount, Clock.systemUTC().instant());
+ }
+
+ private Long getCurrentCount() {
+ return daoV2.listAll().count().block();
+ }
+}
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskAdditionalInformationDTO.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskAdditionalInformationDTO.java
new file mode 100644
index 0000000..acb62c9
--- /dev/null
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskAdditionalInformationDTO.java
@@ -0,0 +1,89 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import java.time.Instant;
+
+import org.apache.james.json.DTOModule;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTO;
+import org.apache.james.server.task.json.dto.AdditionalInformationDTOModule;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class MailboxPathV3MigrationTaskAdditionalInformationDTO implements AdditionalInformationDTO {
+
+ private static MailboxPathV3MigrationTaskAdditionalInformationDTO fromDomainObject(MailboxPathV3Migration.AdditionalInformation additionalInformation, String type) {
+ return new MailboxPathV3MigrationTaskAdditionalInformationDTO(
+ type,
+ additionalInformation.getRemainingCount(),
+ additionalInformation.getInitialCount(),
+ additionalInformation.timestamp()
+ );
+ }
+
+ public static final AdditionalInformationDTOModule<MailboxPathV3Migration.AdditionalInformation, MailboxPathV3MigrationTaskAdditionalInformationDTO> MODULE =
+ DTOModule
+ .forDomainObject(MailboxPathV3Migration.AdditionalInformation.class)
+ .convertToDTO(MailboxPathV3MigrationTaskAdditionalInformationDTO.class)
+ .toDomainObjectConverter(MailboxPathV3MigrationTaskAdditionalInformationDTO::toDomainObject)
+ .toDTOConverter(MailboxPathV3MigrationTaskAdditionalInformationDTO::fromDomainObject)
+ .typeName(MailboxPathV3Migration.TYPE.asString())
+ .withFactory(AdditionalInformationDTOModule::new);
+
+ private final String type;
+ private final long remainingCount;
+ private final long initialCount;
+ private final Instant timestamp;
+
+ public MailboxPathV3MigrationTaskAdditionalInformationDTO(@JsonProperty("type") String type,
+ @JsonProperty("remainingCount") long remainingCount,
+ @JsonProperty("initialCount") long initialCount,
+ @JsonProperty("timestamp") Instant timestamp) {
+ this.type = type;
+ this.remainingCount = remainingCount;
+ this.initialCount = initialCount;
+ this.timestamp = timestamp;
+ }
+
+ public long getRemainingCount() {
+ return remainingCount;
+ }
+
+ public long getInitialCount() {
+ return initialCount;
+ }
+
+ @Override
+ public Instant getTimestamp() {
+ return timestamp;
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+
+ private MailboxPathV3Migration.AdditionalInformation toDomainObject() {
+ return new MailboxPathV3Migration.AdditionalInformation(
+ remainingCount,
+ initialCount,
+ timestamp);
+ }
+}
diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskDTO.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskDTO.java
new file mode 100644
index 0000000..9683ec9
--- /dev/null
+++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskDTO.java
@@ -0,0 +1,59 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import java.util.function.Function;
+
+import org.apache.james.json.DTOModule;
+import org.apache.james.server.task.json.dto.TaskDTO;
+import org.apache.james.server.task.json.dto.TaskDTOModule;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class MailboxPathV3MigrationTaskDTO implements TaskDTO {
+
+ private static MailboxPathV3MigrationTaskDTO fromDomainObject(MailboxPathV3Migration.MailboxPathV3MigrationTask task, String type) {
+ return new MailboxPathV3MigrationTaskDTO(type);
+ }
+
+ public static final Function<MailboxPathV3Migration, TaskDTOModule<MailboxPathV3Migration.MailboxPathV3MigrationTask, MailboxPathV3MigrationTaskDTO>> MODULE = (migration) ->
+ DTOModule
+ .forDomainObject(MailboxPathV3Migration.MailboxPathV3MigrationTask.class)
+ .convertToDTO(MailboxPathV3MigrationTaskDTO.class)
+ .toDomainObjectConverter(dto -> dto.toDomainObject(migration))
+ .toDTOConverter(MailboxPathV3MigrationTaskDTO::fromDomainObject)
+ .typeName(MailboxPathV3Migration.TYPE.asString())
+ .withFactory(TaskDTOModule::new);
+
+ private final String type;
+
+ public MailboxPathV3MigrationTaskDTO(@JsonProperty("type") String type) {
+ this.type = type;
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+
+ private MailboxPathV3Migration.MailboxPathV3MigrationTask toDomainObject(MailboxPathV3Migration migration) {
+ return new MailboxPathV3Migration.MailboxPathV3MigrationTask(migration);
+ }
+}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskSerializationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskSerializationTest.java
new file mode 100644
index 0000000..9bf660a
--- /dev/null
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTaskSerializationTest.java
@@ -0,0 +1,52 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import static org.mockito.Mockito.mock;
+
+import java.time.Instant;
+
+import org.apache.james.JsonSerializationVerifier;
+import org.junit.jupiter.api.Test;
+
+class MailboxPathV3MigrationTaskSerializationTest {
+ private static final Instant TIMESTAMP = Instant.parse("2018-11-13T12:00:55Z");
+ private static final MailboxPathV3Migration MIGRATION = mock(MailboxPathV3Migration.class);
+ private static final MailboxPathV3Migration.MailboxPathV3MigrationTask TASK = new MailboxPathV3Migration.MailboxPathV3MigrationTask(MIGRATION);
+ private static final String SERIALIZED_TASK = "{\"type\": \"cassandra-mailbox-path-v3-migration\"}";
+ private static final MailboxPathV3Migration.AdditionalInformation DETAILS = new MailboxPathV3Migration.AdditionalInformation(42L, 10, TIMESTAMP);
+ private static final String SERIALIZED_ADDITIONAL_INFORMATION = "{\"type\": \"cassandra-mailbox-path-v3-migration\", \"remainingCount\":42,\"initialCount\":10, \"timestamp\":\"2018-11-13T12:00:55Z\"}";
+
+ @Test
+ void taskShouldBeSerializable() throws Exception {
+ JsonSerializationVerifier.dtoModule(MailboxPathV3MigrationTaskDTO.MODULE.apply(MIGRATION))
+ .bean(TASK)
+ .json(SERIALIZED_TASK)
+ .verify();
+ }
+
+ @Test
+ void additionalInformationShouldBeSerializable() throws Exception {
+ JsonSerializationVerifier.dtoModule(MailboxPathV3MigrationTaskAdditionalInformationDTO.MODULE)
+ .bean(DETAILS)
+ .json(SERIALIZED_ADDITIONAL_INFORMATION)
+ .verify();
+ }
+}
diff --git a/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTest.java
new file mode 100644
index 0000000..e0c7108
--- /dev/null
+++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/mail/migration/MailboxPathV3MigrationTest.java
@@ -0,0 +1,100 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import org.apache.james.backends.cassandra.CassandraCluster;
+import org.apache.james.backends.cassandra.CassandraClusterExtension;
+import org.apache.james.backends.cassandra.components.CassandraModule;
+import org.apache.james.backends.cassandra.utils.CassandraUtils;
+import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionModule;
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.cassandra.ids.CassandraId;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxDAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathDAOImpl;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV2DAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMailboxPathV3DAO;
+import org.apache.james.mailbox.cassandra.modules.CassandraAclModule;
+import org.apache.james.mailbox.cassandra.modules.CassandraMailboxModule;
+import org.apache.james.mailbox.model.Mailbox;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UidValidity;
+import org.assertj.core.api.SoftAssertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+class MailboxPathV3MigrationTest {
+
+ private static final MailboxPath MAILBOX_PATH_1 = MailboxPath.forUser(Username.of("bob"), "Important");
+ private static final UidValidity UID_VALIDITY_1 = UidValidity.of(452);
+ private static final CassandraId MAILBOX_ID_1 = CassandraId.timeBased();
+ private static final Mailbox MAILBOX = new Mailbox(MAILBOX_PATH_1, UID_VALIDITY_1, MAILBOX_ID_1);
+
+
+ public static final CassandraModule MODULES = CassandraModule.aggregateModules(
+ CassandraMailboxModule.MODULE,
+ CassandraAclModule.MODULE,
+ CassandraSchemaVersionModule.MODULE);
+
+ @RegisterExtension
+ static CassandraClusterExtension cassandraCluster = new CassandraClusterExtension(MODULES);
+
+ private CassandraMailboxPathDAOImpl daoV1;
+ private CassandraMailboxPathV2DAO daoV2;
+ private CassandraMailboxPathV3DAO daoV3;
+ private CassandraMailboxDAO mailboxDAO;
+
+ @BeforeEach
+ void setUp(CassandraCluster cassandra) {
+ daoV1 = new CassandraMailboxPathDAOImpl(
+ cassandra.getConf(),
+ cassandra.getTypesProvider(),
+ CassandraUtils.WITH_DEFAULT_CONFIGURATION,
+ cassandraCluster.getCassandraConsistenciesConfiguration());
+ daoV2 = new CassandraMailboxPathV2DAO(
+ cassandra.getConf(),
+ CassandraUtils.WITH_DEFAULT_CONFIGURATION,
+ cassandraCluster.getCassandraConsistenciesConfiguration());
+ daoV3 = new CassandraMailboxPathV3DAO(
+ cassandra.getConf(),
+ CassandraUtils.WITH_DEFAULT_CONFIGURATION,
+ cassandraCluster.getCassandraConsistenciesConfiguration());
+
+ mailboxDAO = new CassandraMailboxDAO(
+ cassandra.getConf(),
+ cassandra.getTypesProvider(),
+ cassandraCluster.getCassandraConsistenciesConfiguration());
+ }
+
+ @Test
+ void migrationTaskShouldMoveDataToMostRecentDao() {
+ daoV2.save(MAILBOX_PATH_1, MAILBOX_ID_1).block();
+ mailboxDAO.save(MAILBOX).block();
+
+ new MailboxPathV3Migration(daoV2, daoV3, mailboxDAO).apply();
+
+ SoftAssertions softly = new SoftAssertions();
+ softly.assertThat(daoV1.retrieveId(MAILBOX_PATH_1).blockOptional()).isEmpty();
+ softly.assertThat(daoV2.retrieveId(MAILBOX_PATH_1).blockOptional()).isEmpty();
+ softly.assertThat(daoV3.retrieve(MAILBOX_PATH_1).blockOptional())
+ .contains(MAILBOX);
+ softly.assertAll();
+ }
+}
\ No newline at end of file
diff --git a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/CassandraRoutesModule.java b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/CassandraRoutesModule.java
index 23a4f0a..90e8911 100644
--- a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/CassandraRoutesModule.java
+++ b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/CassandraRoutesModule.java
@@ -26,6 +26,7 @@ import org.apache.james.backends.cassandra.versions.CassandraSchemaVersionManage
import org.apache.james.backends.cassandra.versions.SchemaTransition;
import org.apache.james.backends.cassandra.versions.SchemaVersion;
import org.apache.james.mailbox.cassandra.mail.migration.MailboxPathV2Migration;
+import org.apache.james.mailbox.cassandra.mail.migration.MailboxPathV3Migration;
import org.apache.james.rrt.cassandra.migration.MappingsSourcesMigration;
import org.apache.james.webadmin.Routes;
import org.apache.james.webadmin.routes.CassandraMailboxMergingRoutes;
@@ -40,6 +41,7 @@ import com.google.inject.name.Names;
public class CassandraRoutesModule extends AbstractModule {
private static final SchemaTransition FROM_V5_TO_V6 = SchemaTransition.to(new SchemaVersion(6));
private static final SchemaTransition FROM_V6_TO_V7 = SchemaTransition.to(new SchemaVersion(7));
+ private static final SchemaTransition FROM_V7_TO_V8 = SchemaTransition.to(new SchemaVersion(8));
@Override
protected void configure() {
@@ -57,6 +59,7 @@ public class CassandraRoutesModule extends AbstractModule {
MapBinder<SchemaTransition, Migration> allMigrationClazzBinder = MapBinder.newMapBinder(binder(), SchemaTransition.class, Migration.class);
allMigrationClazzBinder.addBinding(FROM_V5_TO_V6).to(MailboxPathV2Migration.class);
allMigrationClazzBinder.addBinding(FROM_V6_TO_V7).to(MappingsSourcesMigration.class);
+ allMigrationClazzBinder.addBinding(FROM_V7_TO_V8).to(MailboxPathV3Migration.class);
bind(SchemaVersion.class)
.annotatedWith(Names.named(CassandraMigrationService.LATEST_VERSION))
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 22/22: JAMES-3407 Add mailbox.read.repair.chance as
a comment in cassandra.properties examples
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit aeb633ae30b3e4ccd272bd1be284c15ebf583c09
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Oct 8 17:24:08 2020 +0700
JAMES-3407 Add mailbox.read.repair.chance as a comment in cassandra.properties examples
---
.../run/guice/cassandra-ldap/destination/conf/cassandra.properties | 1 +
.../guice/cassandra-rabbitmq-ldap/destination/conf/cassandra.properties | 1 +
.../run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties | 1 +
dockerfiles/run/guice/cassandra/destination/conf/cassandra.properties | 1 +
4 files changed, 4 insertions(+)
diff --git a/dockerfiles/run/guice/cassandra-ldap/destination/conf/cassandra.properties b/dockerfiles/run/guice/cassandra-ldap/destination/conf/cassandra.properties
index c1bbe41..57f1108 100644
--- a/dockerfiles/run/guice/cassandra-ldap/destination/conf/cassandra.properties
+++ b/dockerfiles/run/guice/cassandra-ldap/destination/conf/cassandra.properties
@@ -14,6 +14,7 @@ cassandra.retryConnection.minDelay=5000
## These low level tuning options controls performance related options in Cassandra related code
## The default values are documented here. Understand the effects before editing.
+# mailbox.read.repair.chance=0.1
# mailbox.max.retry.acl=1000
# mailbox.max.retry.modseq=100000
# mailbox.max.retry.uid=100000
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/cassandra.properties b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/cassandra.properties
index e7dff97..dfd3940 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/cassandra.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/cassandra.properties
@@ -24,6 +24,7 @@ cassandra.retryConnection.minDelay=5000
## These low level tuning options controls performance related options in Cassandra related code
## The default values are documented here. Understand the effects before editing.
+# mailbox.read.repair.chance=0.1
# mailbox.max.retry.acl=1000
# mailbox.max.retry.modseq=100000
# mailbox.max.retry.uid=100000
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties
index e7dff97..dfd3940 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/cassandra.properties
@@ -24,6 +24,7 @@ cassandra.retryConnection.minDelay=5000
## These low level tuning options controls performance related options in Cassandra related code
## The default values are documented here. Understand the effects before editing.
+# mailbox.read.repair.chance=0.1
# mailbox.max.retry.acl=1000
# mailbox.max.retry.modseq=100000
# mailbox.max.retry.uid=100000
diff --git a/dockerfiles/run/guice/cassandra/destination/conf/cassandra.properties b/dockerfiles/run/guice/cassandra/destination/conf/cassandra.properties
index 1e1bae2..49a8560 100644
--- a/dockerfiles/run/guice/cassandra/destination/conf/cassandra.properties
+++ b/dockerfiles/run/guice/cassandra/destination/conf/cassandra.properties
@@ -24,6 +24,7 @@ cassandra.retryConnection.minDelay=5000
## These low level tuning options controls performance related options in Cassandra related code
## The default values are documented here. Understand the effects before editing.
+# mailbox.read.repair.chance=0.1
# mailbox.max.retry.acl=1000
# mailbox.max.retry.modseq=100000
# mailbox.max.retry.uid=100000
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[james-project] 01/22: JAMES-3406 Detail more multi-dc setup
Posted by rc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1d5e45986289094dd08f3ed7bd480bcef3e28916
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue Oct 13 08:58:20 2020 +0700
JAMES-3406 Detail more multi-dc setup
---
.../servers/pages/distributed/architecture/consistency-model.adoc | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc b/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc
index e5facc4..a14ad9a 100644
--- a/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc
+++ b/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc
@@ -46,10 +46,12 @@ link:https://docs.datastax.com/en/ddac/doc/datastax_enterprise/production/DDACmu
Cassandra setup is discouraged.
However xref:distributed/configure/cassandra.adoc[this page] enables setting alternative read level,
-which could be acceptable regarding limited requirements.
+which could be acceptable regarding limited requirements. `LOCAL_QUORUM` coupled with `LOCAL_SERIAL`
+is likely the only scalable setup.
-Running the Distributed Server in a multi datacenter setup will likely result either in data loss,
-or very slow operations.
+Running the Distributed Server IMAP server in a multi datacenter setup will likely result either in data loss,
+or very slow operations - as we rely on monotic UID generation, without strong consistency, UIDs could be allocated
+several times. Other protocols, like JMAP, suffer less from such limitations.
=== ElasticSearch consistency model
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org