You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by rc...@apache.org on 2021/01/18 03:25:20 UTC
[james-project] 03/03: JAMES-3202 Allow Reindexing without cleanup
This is an automated email from the ASF dual-hosted git repository.
rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit b1eab55728aecb0c13ab2bdba5be0de434030ee4
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Dec 31 16:54:14 2020 +0700
JAMES-3202 Allow Reindexing without cleanup
Avoids running delete-by-query against empty indexes:
- Performance wise this unneeded operation is harmful
- Caused some index-out-of-bound exceptions on one of our indexes (error could clearly be linked to this)
---
.../apache/james/mailbox/indexer/ReIndexer.java | 1 +
.../mailbox/tools/indexer/ReIndexerPerformer.java | 10 +-
.../james/webadmin/routes/MailboxesRoutesTest.java | 121 +++++++++++++++++++++
src/site/markdown/server/manage-webadmin.md | 20 +++-
4 files changed, 144 insertions(+), 8 deletions(-)
diff --git a/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java b/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
index e953059..bf8bde7 100644
--- a/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
+++ b/mailbox/api/src/main/java/org/apache/james/mailbox/indexer/ReIndexer.java
@@ -69,6 +69,7 @@ public interface ReIndexer {
public enum Mode {
REBUILD_ALL("rebuildAll"),
+ REBUILD_ALL_NO_CLEANUP("rebuildAllNoCleanup"),
FIX_OUTDATED("fixOutdated");
private final String value;
diff --git a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
index 8343f32..c51338c 100644
--- a/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
+++ b/mailbox/tools/indexer/src/main/java/org/apache/mailbox/tools/indexer/ReIndexerPerformer.java
@@ -271,10 +271,14 @@ public class ReIndexerPerformer {
}
private Mono<Void> updateSearchIndex(Mailbox mailbox, MailboxSession mailboxSession, RunningOptions runningOptions) {
- if (runningOptions.getMode() == RunningOptions.Mode.REBUILD_ALL) {
- return messageSearchIndex.deleteAll(mailboxSession, mailbox.getMailboxId());
+ switch (runningOptions.getMode()) {
+ case REBUILD_ALL:
+ return messageSearchIndex.deleteAll(mailboxSession, mailbox.getMailboxId());
+ case REBUILD_ALL_NO_CLEANUP:
+ case FIX_OUTDATED:
+ default:
+ return Mono.empty();
}
- return Mono.empty();
}
private Mono<Task.Result> reIndexMessages(Flux<Either<Failure, ReIndexingEntry>> entriesToIndex, RunningOptions runningOptions, ReprocessingContext reprocessingContext) {
diff --git a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
index ea4afe3..d53f165 100644
--- a/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
+++ b/server/protocols/webadmin/webadmin-mailbox/src/test/java/org/apache/james/webadmin/routes/MailboxesRoutesTest.java
@@ -28,6 +28,7 @@ import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.notNullValue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.never;
import static org.mockito.Mockito.reset;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
@@ -104,6 +105,7 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.mockito.ArgumentCaptor;
+import com.github.steveash.guavate.Guavate;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
@@ -432,6 +434,60 @@ class MailboxesRoutesTest {
}
@Test
+ void fullReprocessingShouldAcceptRebuildAllNoCleanupMode() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+ mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession);
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ Flags newFlags = new Flags(Flags.Flag.DRAFT);
+ UpdatedFlags updatedFlags = UpdatedFlags.builder()
+ .uid(result.getUid())
+ .modSeq(messages.get(0).getModSeq())
+ .oldFlags(new Flags())
+ .newFlags(newFlags)
+ .build();
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.update(systemSession, mailbox.getMailboxId(), ImmutableList.of(updatedFlags)).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=rebuildAllNoCleanup")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await")
+ .then()
+ .body("status", is("completed"))
+ .body("taskId", is(notNullValue()))
+ .body("type", is(FullReindexingTask.FULL_RE_INDEXING.asString()))
+ .body("additionalInformation.successfullyReprocessedMailCount", is(2))
+ .body("additionalInformation.failedReprocessedMailCount", is(0))
+ .body("additionalInformation.runningOptions.messagesPerSecond", is(50))
+ .body("additionalInformation.runningOptions.mode", is("REBUILD_ALL_NO_CLEANUP"))
+ .body("startedDate", is(notNullValue()))
+ .body("submitDate", is(notNullValue()))
+ .body("completedDate", is(notNullValue()));
+
+ // verify that deleteAll on index never got called with rebuildAllNoCleanup mode
+ verify(searchIndex, never()).deleteAll(any(MailboxSession.class), any(MailboxId.class));
+ }
+
+ @Test
void fullReprocessingWithCorrectModeShouldFixInconsistenciesInES() throws Exception {
MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
@@ -473,6 +529,71 @@ class MailboxesRoutesTest {
}
@Test
+ void fullReprocessingNoCleanupShouldNoopWhenNoInconsistencies() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=rebuildAllNoCleanup")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Test
+ void fullReprocessingNoCleanupShouldSolveInconsistencies() throws Exception {
+ MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
+ MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
+ Mailbox mailbox = mailboxManager.getMailbox(mailboxId, systemSession).getMailboxEntity();
+
+ ComposedMessageId result = mailboxManager.getMailbox(INBOX, systemSession)
+ .appendMessage(
+ MessageManager.AppendCommand.builder().build("header: value\r\n\r\nbody"),
+ systemSession)
+ .getId();
+
+ Flags initialFlags = searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block();
+
+ List<MessageResult> messages = messageIdManager.getMessages(ImmutableList.of(result.getMessageId()), FetchGroup.MINIMAL, systemSession);
+
+ // We update on the searchIndex level to try to create inconsistencies
+ searchIndex.delete(systemSession, mailbox.getMailboxId(),
+ messages.stream()
+ .map(MessageResult::getUid)
+ .collect(Guavate.toImmutableList()))
+ .block();
+
+ String taskId = with()
+ .post("/mailboxes?task=reIndex&mode=rebuildAllNoCleanup")
+ .jsonPath()
+ .get("taskId");
+
+ given()
+ .basePath(TasksRoutes.BASE)
+ .when()
+ .get(taskId + "/await");
+
+ assertThat(searchIndex.retrieveIndexedFlags(mailbox, result.getUid()).block())
+ .isEqualTo(initialFlags);
+ }
+
+ @Test
void fullReprocessingWithCorrectModeShouldNotChangeDocumentsInESWhenNoInconsistencies() throws Exception {
MailboxSession systemSession = mailboxManager.createSystemSession(USERNAME);
MailboxId mailboxId = mailboxManager.createMailbox(INBOX, systemSession).get();
diff --git a/src/site/markdown/server/manage-webadmin.md b/src/site/markdown/server/manage-webadmin.md
index a386681..e99f476 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -611,7 +611,9 @@ This optional parameter must have a strictly positive integer as a value and be
An admin can also specify the reindexing mode it wants to use when running the task:
- `mode` the reindexing mode used. There are 2 modes for the moment:
- - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `rebuildAll` allows to rebuild all indexes. It drops indexed entries prior reindexing. This is the default mode.
+ - `rebuildAllNoCleanup` allows to rebuild all indexes. It skips the cleanup phase thus will not remove evicted entries
+ upon reindex. However it yields better performances on a known to be empty index.
- `fixOutdated` will check for outdated indexed document and reindex only those.
This optional parameter must be passed as query parameter.
@@ -665,7 +667,9 @@ This optional parameter must have a strictly positive integer as a value and be
An admin can also specify the reindexing mode it wants to use when running the task:
- `mode` the reindexing mode used. There are 2 modes for the moment:
- - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `rebuildAll` allows to rebuild all indexes. It drops indexed entries prior reindexing. This is the default mode.
+ - `rebuildAllNoCleanup` allows to rebuild all indexes. It skips the cleanup phase thus will not remove evicted entries
+ upon reindex. However it yields better performances on a known to be empty index.
- `fixOutdated` will check for outdated indexed document and reindex only those.
This optional parameter must be passed as query parameter.
@@ -725,7 +729,9 @@ This optional parameter must have a strictly positive integer as a value and be
An admin can also specify the reindexing mode it wants to use when running the task:
- `mode` the reindexing mode used. There are 2 modes for the moment:
- - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `rebuildAll` allows to rebuild all indexes. It drops indexed entries prior reindexing. This is the default mode.
+ - `rebuildAllNoCleanup` allows to rebuild all indexes. It skips the cleanup phase thus will not remove evicted entries
+ upon reindex. However it yields better performances on a known to be empty index.
- `fixOutdated` will check for outdated indexed document and reindex only those.
This optional parameter must be passed as query parameter.
@@ -876,7 +882,9 @@ This optional parameter must have a strictly positive integer as a value and be
An admin can also specify the reindexing mode it wants to use when running the task:
- `mode` the reindexing mode used. There are 2 modes for the moment:
- - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `rebuildAll` allows to rebuild all indexes. It drops indexed entries prior reindexing. This is the default mode.
+ - `rebuildAllNoCleanup` allows to rebuild all indexes. It skips the cleanup phase thus will not remove evicted entries
+ upon reindex. However it yields better performances on a known to be empty index.
- `fixOutdated` will check for outdated indexed document and reindex only those.
This optional parameter must be passed as query parameter.
@@ -1078,7 +1086,9 @@ This optional parameter must have a strictly positive integer as a value and be
An admin can also specify the reindexing mode it wants to use when running the task:
- `mode` the reindexing mode used. There are 2 modes for the moment:
- - `rebuildAll` allows to rebuild all indexes. This is the default mode.
+ - `rebuildAll` allows to rebuild all indexes. It drops indexed entries prior reindexing. This is the default mode.
+ - `rebuildAllNoCleanup` allows to rebuild all indexes. It skips the cleanup phase thus will not remove evicted entries
+ upon reindex. However it yields better performances on a known to be empty index.
- `fixOutdated` will check for outdated indexed document and reindex only those.
This optional parameter must be passed as query parameter.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org