You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "chibenwa (via GitHub)" <gi...@apache.org> on 2024/02/18 20:55:05 UTC
[PR] Do not index body [james-project]
chibenwa opened a new pull request, #2018:
URL: https://github.com/apache/james-project/pull/2018
This relax pressure on the AES blob store and APPEND
Also this is helpfull upon mass data ingestion.
TODO make this configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1500455955
##########
mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchNoIndexBodyIntegrationTest.java:
##########
@@ -0,0 +1,197 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.opensearch;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Durations.ONE_HUNDRED_MILLISECONDS;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.ZoneId;
+import java.util.UUID;
+
+import org.apache.james.backends.opensearch.DockerOpenSearchExtension;
+import org.apache.james.backends.opensearch.IndexName;
+import org.apache.james.backends.opensearch.OpenSearchIndexer;
+import org.apache.james.backends.opensearch.ReactorOpenSearchClient;
+import org.apache.james.backends.opensearch.ReadAliasName;
+import org.apache.james.backends.opensearch.WriteAliasName;
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.inmemory.InMemoryMailboxManager;
+import org.apache.james.mailbox.inmemory.InMemoryMessageId;
+import org.apache.james.mailbox.inmemory.manager.InMemoryIntegrationResources;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.Mailbox;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.SearchQuery;
+import org.apache.james.mailbox.opensearch.events.OpenSearchListeningMessageSearchIndex;
+import org.apache.james.mailbox.opensearch.json.MessageToOpenSearchJson;
+import org.apache.james.mailbox.opensearch.query.CriterionConverter;
+import org.apache.james.mailbox.opensearch.query.QueryConverter;
+import org.apache.james.mailbox.opensearch.search.OpenSearchSearcher;
+import org.apache.james.mailbox.store.StoreMessageManager;
+import org.apache.james.mailbox.store.search.MessageSearchIndex;
+import org.apache.james.mailbox.tika.TikaConfiguration;
+import org.apache.james.mailbox.tika.TikaExtension;
+import org.apache.james.mailbox.tika.TikaHttpClientImpl;
+import org.apache.james.mailbox.tika.TikaTextExtractor;
+import org.apache.james.metrics.tests.RecordingMetricFactory;
+import org.apache.james.mime4j.dom.Message;
+import org.awaitility.Awaitility;
+import org.awaitility.Durations;
+import org.awaitility.core.ConditionFactory;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.opensearch.client.opensearch._types.query_dsl.Query;
+import org.opensearch.client.opensearch._types.query_dsl.QueryBuilders;
+import org.opensearch.client.opensearch.core.SearchRequest;
+
+import com.google.common.collect.ImmutableSet;
+
+import reactor.core.publisher.Mono;
+
+public class OpenSearchNoIndexBodyIntegrationTest {
+ static final int SEARCH_SIZE = 1;
+ private static final Username USERNAME = Username.of("user");
+ private static final ConditionFactory CALMLY_AWAIT = Awaitility
+ .with().pollInterval(ONE_HUNDRED_MILLISECONDS)
+ .and().pollDelay(ONE_HUNDRED_MILLISECONDS)
+ .await();
+
+ @RegisterExtension
+ static TikaExtension tika = new TikaExtension();
+
+ @RegisterExtension
+ static DockerOpenSearchExtension openSearch = new DockerOpenSearchExtension(DockerOpenSearchExtension.CleanupStrategy.NONE);
+
+ TikaTextExtractor textExtractor;
+ ReactorOpenSearchClient client;
+ private InMemoryMailboxManager storeMailboxManager;
+ private MessageSearchIndex messageSearchIndex;
+ private IndexName indexName;
+ protected MailboxSession session;
+ private Mailbox mailbox;
+ private MailboxPath inboxPath;
+
+
+ @BeforeEach
+ void setUp() throws Exception {
+ textExtractor = new TikaTextExtractor(new RecordingMetricFactory(),
+ new TikaHttpClientImpl(TikaConfiguration.builder()
+ .host(tika.getIp())
+ .port(tika.getPort())
+ .timeoutInMillis(tika.getTimeoutInMillis())
+ .build()));
+ client = openSearch.getDockerOpenSearch().clientProvider().get();
+
+ InMemoryMessageId.Factory messageIdFactory = new InMemoryMessageId.Factory();
+ MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
+ ReadAliasName readAliasName = new ReadAliasName(UUID.randomUUID().toString());
+ WriteAliasName writeAliasName = new WriteAliasName(UUID.randomUUID().toString());
+ indexName = new IndexName(UUID.randomUUID().toString());
+ MailboxIndexCreationUtil.prepareClient(client, readAliasName, writeAliasName, indexName,
+ openSearch.getDockerOpenSearch().configuration());
+
+ InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
+ .preProvisionnedFakeAuthenticator()
+ .fakeAuthorizator()
+ .inVmEventBus()
+ .defaultAnnotationLimits()
+ .defaultMessageParser()
+ .listeningSearchIndex(preInstanciationStage -> new OpenSearchListeningMessageSearchIndex(
+ preInstanciationStage.getMapperFactory(),
+ ImmutableSet.of(),
+ new OpenSearchIndexer(client, writeAliasName),
+ new OpenSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE, readAliasName, routingKeyFactory),
+ new MessageToOpenSearchJson(textExtractor, ZoneId.of("Europe/Paris"), IndexAttachments.YES, IndexHeaders.YES),
+ preInstanciationStage.getSessionProvider(), routingKeyFactory, messageIdFactory,
+ OpenSearchMailboxConfiguration.builder().indexBody(IndexBody.NO).build(), new RecordingMetricFactory()))
+ .noPreDeletionHooks()
+ .storeQuotaManager()
+ .build();
+
+ storeMailboxManager = resources.getMailboxManager();
+ messageSearchIndex = resources.getSearchIndex();
+
+ session = storeMailboxManager.createSystemSession(USERNAME);
+ inboxPath = MailboxPath.inbox(USERNAME);
+ storeMailboxManager.createMailbox(inboxPath, session);
+
+ StoreMessageManager inboxMessageManager = (StoreMessageManager) storeMailboxManager.getMailbox(inboxPath, session);
+ mailbox = inboxMessageManager.getMailboxEntity();
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ client.close();
+ }
+
+ @Test
+ void searchingByBodyContentShouldNotReturnMessageWhenNoIndexBody() throws Exception {
+ addMessage(session, inboxPath).block();
+
+ awaitForOpenSearch(QueryBuilders.matchAll().build()._toQuery(), 1L);
+
+ SearchQuery searchQuery = SearchQuery.of(SearchQuery.bodyContains("Hello"));
+
+ assertThat(messageSearchIndex.search(session, mailbox, searchQuery).collectList().block())
+ .isEmpty();
+ }
+
+ @Test
+ void searchingAllShoulReturnMessageWhenNoIndexBody() throws Exception {
+ ComposedMessageId composedMessageId = addMessage(session, inboxPath).block();
+
+ awaitForOpenSearch(QueryBuilders.matchAll().build()._toQuery(), 1L);
+
+ SearchQuery searchQuery = SearchQuery.of(SearchQuery.bodyContains("Hello"));
Review Comment:
```suggestion
SearchQuery searchQuery = SearchQuery.of(SearchQuery.all());
```
To match the test name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1952218252
> But the non indexing of the body should not be enabled by default
Of course.
I'm asking for an emergency safety trigger.
Not a default value.
Like acl.enabled in casandra.properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1500456865
##########
src/site/xdoc/server/config-opensearch.xml:
##########
@@ -122,6 +122,8 @@
<dd>When set to true, James will attempt to reindex from the indexed message when moved.
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source).
Default to false.</dd>
+ <dt><strong>opensearch.indexBody</strong></dt>
+ <dd>Indicates if you wish to index body or not (default: true).</dd>
Review Comment:
```suggestion
<dd>Indicates if you wish to index body or not (default: true). This can be used to decrease the performance cost associated with indexing.</dd>
```
##########
server/apps/distributed-app/docs/modules/ROOT/pages/configure/opensearch.adoc:
##########
@@ -96,6 +96,9 @@ turning off headers indexing result in non-strict compliance with the IMAP / JMA
| When set to true, James will attempt to reindex from the indexed message when moved.
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source)
Default to false.
+
+| opensearch.indexBody
+| Indicates if you wish to index body or not (default: true).
Review Comment:
```suggestion
| Indicates if you wish to index body or not (default: true). This can be used to decrease the performance cost associated with indexing.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1500462621
##########
mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchNoIndexBodyIntegrationTest.java:
##########
@@ -0,0 +1,197 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.opensearch;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Durations.ONE_HUNDRED_MILLISECONDS;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.ZoneId;
+import java.util.UUID;
+
+import org.apache.james.backends.opensearch.DockerOpenSearchExtension;
+import org.apache.james.backends.opensearch.IndexName;
+import org.apache.james.backends.opensearch.OpenSearchIndexer;
+import org.apache.james.backends.opensearch.ReactorOpenSearchClient;
+import org.apache.james.backends.opensearch.ReadAliasName;
+import org.apache.james.backends.opensearch.WriteAliasName;
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.inmemory.InMemoryMailboxManager;
+import org.apache.james.mailbox.inmemory.InMemoryMessageId;
+import org.apache.james.mailbox.inmemory.manager.InMemoryIntegrationResources;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.Mailbox;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.SearchQuery;
+import org.apache.james.mailbox.opensearch.events.OpenSearchListeningMessageSearchIndex;
+import org.apache.james.mailbox.opensearch.json.MessageToOpenSearchJson;
+import org.apache.james.mailbox.opensearch.query.CriterionConverter;
+import org.apache.james.mailbox.opensearch.query.QueryConverter;
+import org.apache.james.mailbox.opensearch.search.OpenSearchSearcher;
+import org.apache.james.mailbox.store.StoreMessageManager;
+import org.apache.james.mailbox.store.search.MessageSearchIndex;
+import org.apache.james.mailbox.tika.TikaConfiguration;
+import org.apache.james.mailbox.tika.TikaExtension;
+import org.apache.james.mailbox.tika.TikaHttpClientImpl;
+import org.apache.james.mailbox.tika.TikaTextExtractor;
+import org.apache.james.metrics.tests.RecordingMetricFactory;
+import org.apache.james.mime4j.dom.Message;
+import org.awaitility.Awaitility;
+import org.awaitility.Durations;
+import org.awaitility.core.ConditionFactory;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.opensearch.client.opensearch._types.query_dsl.Query;
+import org.opensearch.client.opensearch._types.query_dsl.QueryBuilders;
+import org.opensearch.client.opensearch.core.SearchRequest;
+
+import com.google.common.collect.ImmutableSet;
+
+import reactor.core.publisher.Mono;
+
+public class OpenSearchNoIndexBodyIntegrationTest {
+ static final int SEARCH_SIZE = 1;
+ private static final Username USERNAME = Username.of("user");
+ private static final ConditionFactory CALMLY_AWAIT = Awaitility
+ .with().pollInterval(ONE_HUNDRED_MILLISECONDS)
+ .and().pollDelay(ONE_HUNDRED_MILLISECONDS)
+ .await();
+
+ @RegisterExtension
+ static TikaExtension tika = new TikaExtension();
+
+ @RegisterExtension
+ static DockerOpenSearchExtension openSearch = new DockerOpenSearchExtension(DockerOpenSearchExtension.CleanupStrategy.NONE);
+
+ TikaTextExtractor textExtractor;
+ ReactorOpenSearchClient client;
+ private InMemoryMailboxManager storeMailboxManager;
+ private MessageSearchIndex messageSearchIndex;
+ private IndexName indexName;
+ protected MailboxSession session;
+ private Mailbox mailbox;
+ private MailboxPath inboxPath;
+
+
+ @BeforeEach
+ void setUp() throws Exception {
+ textExtractor = new TikaTextExtractor(new RecordingMetricFactory(),
+ new TikaHttpClientImpl(TikaConfiguration.builder()
+ .host(tika.getIp())
+ .port(tika.getPort())
+ .timeoutInMillis(tika.getTimeoutInMillis())
+ .build()));
+ client = openSearch.getDockerOpenSearch().clientProvider().get();
+
+ InMemoryMessageId.Factory messageIdFactory = new InMemoryMessageId.Factory();
+ MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
+ ReadAliasName readAliasName = new ReadAliasName(UUID.randomUUID().toString());
+ WriteAliasName writeAliasName = new WriteAliasName(UUID.randomUUID().toString());
+ indexName = new IndexName(UUID.randomUUID().toString());
+ MailboxIndexCreationUtil.prepareClient(client, readAliasName, writeAliasName, indexName,
+ openSearch.getDockerOpenSearch().configuration());
+
+ InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
+ .preProvisionnedFakeAuthenticator()
+ .fakeAuthorizator()
+ .inVmEventBus()
+ .defaultAnnotationLimits()
+ .defaultMessageParser()
+ .listeningSearchIndex(preInstanciationStage -> new OpenSearchListeningMessageSearchIndex(
+ preInstanciationStage.getMapperFactory(),
+ ImmutableSet.of(),
+ new OpenSearchIndexer(client, writeAliasName),
+ new OpenSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE, readAliasName, routingKeyFactory),
+ new MessageToOpenSearchJson(textExtractor, ZoneId.of("Europe/Paris"), IndexAttachments.YES, IndexHeaders.YES),
+ preInstanciationStage.getSessionProvider(), routingKeyFactory, messageIdFactory,
+ OpenSearchMailboxConfiguration.builder().indexBody(IndexBody.NO).build(), new RecordingMetricFactory()))
+ .noPreDeletionHooks()
+ .storeQuotaManager()
+ .build();
+
+ storeMailboxManager = resources.getMailboxManager();
+ messageSearchIndex = resources.getSearchIndex();
+
+ session = storeMailboxManager.createSystemSession(USERNAME);
+ inboxPath = MailboxPath.inbox(USERNAME);
+ storeMailboxManager.createMailbox(inboxPath, session);
+
+ StoreMessageManager inboxMessageManager = (StoreMessageManager) storeMailboxManager.getMailbox(inboxPath, session);
+ mailbox = inboxMessageManager.getMailboxEntity();
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ client.close();
+ }
+
+ @Test
+ void searchingByBodyContentShouldNotReturnMessageWhenNoIndexBody() throws Exception {
+ addMessage(session, inboxPath).block();
+
+ awaitForOpenSearch(QueryBuilders.matchAll().build()._toQuery(), 1L);
+
+ SearchQuery searchQuery = SearchQuery.of(SearchQuery.bodyContains("Hello"));
+
+ assertThat(messageSearchIndex.search(session, mailbox, searchQuery).collectList().block())
+ .isEmpty();
Review Comment:
Memory backend is not compliant with FetchTyle and always return the full message. CF `InMemoryMessageMapper::findInMailbox`.
We should cary other the indexed body config deep into `MimePartParser::parse` and use `message.getHeaderContent()` instead of `message.getFullContent()` in order to make it work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1500453764
##########
mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/events/OpenSearchListeningMessageSearchIndex.java:
##########
@@ -296,7 +299,14 @@ private Mono<Void> handleMailboxEvent(Event event, MailboxSession session, Mailb
private Mono<Void> processAddedEvent(MailboxSession session, MailboxEvents.Added addedEvent, MailboxId mailboxId) {
return factory.getMailboxMapper(session)
.findMailboxById(mailboxId)
- .flatMap(mailbox -> handleAdded(session, mailbox, addedEvent));
+ .flatMap(mailbox -> processAdded(session, mailbox, addedEvent));
+ }
+
+ private Mono<Void> processAdded(MailboxSession session, Mailbox mailbox, MailboxEvents.Added addedEvent) {
+ if (IndexBody.YES.equals(indexBody)) {
+ return handleAdded(session, mailbox, addedEvent, FetchType.FULL);
+ }
+ return handleAdded(session, mailbox, addedEvent, FetchType.HEADERS);
Review Comment:
Only choose the FetchType amd avoid code duplication please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa merged PR #2018:
URL: https://github.com/apache/james-project/pull/2018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1500446233
##########
mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchNoIndexBodyIntegrationTest.java:
##########
@@ -0,0 +1,197 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.opensearch;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Durations.ONE_HUNDRED_MILLISECONDS;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.ZoneId;
+import java.util.UUID;
+
+import org.apache.james.backends.opensearch.DockerOpenSearchExtension;
+import org.apache.james.backends.opensearch.IndexName;
+import org.apache.james.backends.opensearch.OpenSearchIndexer;
+import org.apache.james.backends.opensearch.ReactorOpenSearchClient;
+import org.apache.james.backends.opensearch.ReadAliasName;
+import org.apache.james.backends.opensearch.WriteAliasName;
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.inmemory.InMemoryMailboxManager;
+import org.apache.james.mailbox.inmemory.InMemoryMessageId;
+import org.apache.james.mailbox.inmemory.manager.InMemoryIntegrationResources;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.Mailbox;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.SearchQuery;
+import org.apache.james.mailbox.opensearch.events.OpenSearchListeningMessageSearchIndex;
+import org.apache.james.mailbox.opensearch.json.MessageToOpenSearchJson;
+import org.apache.james.mailbox.opensearch.query.CriterionConverter;
+import org.apache.james.mailbox.opensearch.query.QueryConverter;
+import org.apache.james.mailbox.opensearch.search.OpenSearchSearcher;
+import org.apache.james.mailbox.store.StoreMessageManager;
+import org.apache.james.mailbox.store.search.MessageSearchIndex;
+import org.apache.james.mailbox.tika.TikaConfiguration;
+import org.apache.james.mailbox.tika.TikaExtension;
+import org.apache.james.mailbox.tika.TikaHttpClientImpl;
+import org.apache.james.mailbox.tika.TikaTextExtractor;
+import org.apache.james.metrics.tests.RecordingMetricFactory;
+import org.apache.james.mime4j.dom.Message;
+import org.awaitility.Awaitility;
+import org.awaitility.Durations;
+import org.awaitility.core.ConditionFactory;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.opensearch.client.opensearch._types.query_dsl.Query;
+import org.opensearch.client.opensearch._types.query_dsl.QueryBuilders;
+import org.opensearch.client.opensearch.core.SearchRequest;
+
+import com.google.common.collect.ImmutableSet;
+
+import reactor.core.publisher.Mono;
+
+public class OpenSearchNoIndexBodyIntegrationTest {
+ static final int SEARCH_SIZE = 1;
+ private static final Username USERNAME = Username.of("user");
+ private static final ConditionFactory CALMLY_AWAIT = Awaitility
+ .with().pollInterval(ONE_HUNDRED_MILLISECONDS)
+ .and().pollDelay(ONE_HUNDRED_MILLISECONDS)
+ .await();
+
+ @RegisterExtension
+ static TikaExtension tika = new TikaExtension();
+
+ @RegisterExtension
+ static DockerOpenSearchExtension openSearch = new DockerOpenSearchExtension(DockerOpenSearchExtension.CleanupStrategy.NONE);
+
+ TikaTextExtractor textExtractor;
+ ReactorOpenSearchClient client;
+ private InMemoryMailboxManager storeMailboxManager;
+ private MessageSearchIndex messageSearchIndex;
+ private IndexName indexName;
+ protected MailboxSession session;
+ private Mailbox mailbox;
+ private MailboxPath inboxPath;
+
+
+ @BeforeEach
+ void setUp() throws Exception {
+ textExtractor = new TikaTextExtractor(new RecordingMetricFactory(),
+ new TikaHttpClientImpl(TikaConfiguration.builder()
+ .host(tika.getIp())
+ .port(tika.getPort())
+ .timeoutInMillis(tika.getTimeoutInMillis())
+ .build()));
+ client = openSearch.getDockerOpenSearch().clientProvider().get();
+
+ InMemoryMessageId.Factory messageIdFactory = new InMemoryMessageId.Factory();
+ MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
+ ReadAliasName readAliasName = new ReadAliasName(UUID.randomUUID().toString());
+ WriteAliasName writeAliasName = new WriteAliasName(UUID.randomUUID().toString());
+ indexName = new IndexName(UUID.randomUUID().toString());
+ MailboxIndexCreationUtil.prepareClient(client, readAliasName, writeAliasName, indexName,
+ openSearch.getDockerOpenSearch().configuration());
+
+ InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
+ .preProvisionnedFakeAuthenticator()
+ .fakeAuthorizator()
+ .inVmEventBus()
+ .defaultAnnotationLimits()
+ .defaultMessageParser()
+ .listeningSearchIndex(preInstanciationStage -> new OpenSearchListeningMessageSearchIndex(
+ preInstanciationStage.getMapperFactory(),
+ ImmutableSet.of(),
+ new OpenSearchIndexer(client, writeAliasName),
+ new OpenSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE, readAliasName, routingKeyFactory),
+ new MessageToOpenSearchJson(textExtractor, ZoneId.of("Europe/Paris"), IndexAttachments.YES, IndexHeaders.YES),
+ preInstanciationStage.getSessionProvider(), routingKeyFactory, messageIdFactory,
+ OpenSearchMailboxConfiguration.builder().indexBody(IndexBody.NO).build(), new RecordingMetricFactory()))
+ .noPreDeletionHooks()
+ .storeQuotaManager()
+ .build();
+
+ storeMailboxManager = resources.getMailboxManager();
+ messageSearchIndex = resources.getSearchIndex();
+
+ session = storeMailboxManager.createSystemSession(USERNAME);
+ inboxPath = MailboxPath.inbox(USERNAME);
+ storeMailboxManager.createMailbox(inboxPath, session);
+
+ StoreMessageManager inboxMessageManager = (StoreMessageManager) storeMailboxManager.getMailbox(inboxPath, session);
+ mailbox = inboxMessageManager.getMailboxEntity();
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ client.close();
+ }
+
+ @Test
+ void searchingByBodyContentShouldNotReturnMessageWhenNoIndexBody() throws Exception {
+ addMessage(session, inboxPath).block();
+
+ awaitForOpenSearch(QueryBuilders.matchAll().build()._toQuery(), 1L);
+
+ SearchQuery searchQuery = SearchQuery.of(SearchQuery.bodyContains("Hello"));
+
+ assertThat(messageSearchIndex.search(session, mailbox, searchQuery).collectList().block())
+ .isEmpty();
Review Comment:
Well fails:
```
java.lang.AssertionError:
Expecting empty but was: [MessageUid{uid=1}]
```
Need to debug that, not sure where I messed up...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1963076095
https://issues.apache.org/jira/browse/JAMES-4010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael commented on code in PR #2018:
URL: https://github.com/apache/james-project/pull/2018#discussion_r1494069538
##########
mailbox/store/src/main/java/org/apache/james/mailbox/store/search/ListeningMessageSearchIndex.java:
##########
@@ -132,7 +132,7 @@ protected Mono<Void> handleAdded(MailboxSession session, Mailbox mailbox, Added
private Flux<MailboxMessage> retrieveMailboxMessages(MailboxSession session, Mailbox mailbox, MessageRange range) {
return factory.getMessageMapper(session)
- .findInMailboxReactive(mailbox, range, FetchType.FULL, UNLIMITED);
+ .findInMailboxReactive(mailbox, range, FetchType.HEADERS, UNLIMITED);
Review Comment:
Wouldn't that make the search on the mail body unavailable though? I don't see why a client using James would find that acceptable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1961382503
```
12:45:54,704 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.3.0:check (check-style) on project apache-james-mailbox-opensearch: You have 1 Checkstyle violation. -> [Help 1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1952138840
Unavailable on body but you get all the other search properties.
This is SUPER handy as a "panic fix" or for people wanting to compromise on the cost of search.
(you can still search from, to cc, bcc, subject, dates, etc...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] Do not index body [james-project]
Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1952148847
I get you can search on other header fields. I can admit maybe in some cases it could be wanted. But the non indexing of the body should not be enabled by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org