You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "vttranlina (via GitHub)" <gi...@apache.org> on 2024/10/30 01:52:06 UTC

[PR] WIP - JAMES-4082 Lucene search attachment [james-project]

vttranlina opened a new pull request, #2476:
URL: https://github.com/apache/james-project/pull/2476

   resolve https://github.com/linagora/james-project/issues/5306
   jira: https://issues.apache.org/jira/browse/JAMES-4082
   
   My idea:
   
   - In the `opensearch` module, it contains almost api for indexing, that can be reused for `lucene` 
   eg: 
   
   ```
   import org.apache.james.mailbox.opensearch.json.HeaderCollection;
   import org.apache.james.mailbox.opensearch.json.MimePart;
   import org.apache.james.mailbox.opensearch.json.MimePartParser;
   ```
   With the supported API, using `TextExtractor` for attachment indexing will be easier.
   
   - In this PR, I also reworked and cleaned up the `LuceneMessageSearchIndex#createMessageDocument` method, which has had excessive boilerplate since 2015.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael merged PR #2476:
URL: https://github.com/apache/james-project/pull/2476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827353799


##########
server/apps/jpa-app/src/main/java/org/apache/james/JPAJamesServerMain.java:
##########
@@ -82,15 +84,19 @@ public class JPAJamesServerMain implements JamesServerMain {
         new SMTPServerModule(),
         WEBADMIN);
 
+    private static final Module SEARCH_MODULE = Modules.combine(
+        new LuceneSearchMailboxModule(),
+        binder -> binder.bind(TextExtractor.class).toInstance(new DefaultTextExtractor()));

Review Comment:
   `JsoupTextExtractor` is more powerful?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827159289


##########
mailbox/lucene/src/test/java/org/apache/james/mailbox/lucene/search/LuceneMemorySearchHighLighterTest.java:
##########
@@ -112,4 +126,46 @@ public void applyRightsCommand(MailboxId mailboxId, Username owner, Username del
             MailboxACL.command().forUser(delegated).rights(MailboxACL.FULL_RIGHTS).asAddition(),
             session(owner))).run();
     }
+
+
+    @Test
+    void shouldHighlightAttachmentTextContentWhenTextBodyDoesNotMatch() throws Exception {

Review Comment:
   Any reason why it is not in the contract?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827158945


##########
mailbox/lucene/src/test/java/org/apache/james/mailbox/lucene/search/LuceneFSSearchHighLighterTest.java:
##########
@@ -65,7 +66,7 @@ public void setUp(@TempDir Path tempPath) throws Exception {
             .listeningSearchIndex(Throwing.function(preInstanciationStage -> new LuceneMessageSearchIndex(
                 preInstanciationStage.getMapperFactory(), new InMemoryId.Factory(), FSDirectory.open(tempPath),
                 messageIdFactory,
-                preInstanciationStage.getSessionProvider())))
+                preInstanciationStage.getSessionProvider(), new JsoupTextExtractor())))

Review Comment:
   Cool but it will likely create Spring issues, correct?
   
   We need to fixup spring injects as well (inject JsoupTextExtractor is enough?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] WIP - JAMES-4082 Lucene search attachment [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1822163647


##########
mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneIndexableDocument.java:
##########
@@ -0,0 +1,274 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one   *
+ * or more contributor license agreements.  See the NOTICE file *
+ * distributed with this work for additional information        *
+ * regarding copyright ownership.  The ASF licenses this file   *
+ * to you under the Apache License, Version 2.0 (the            *
+ * "License"); you may not use this file except in compliance   *
+ * with the License.  You may obtain a copy of the License at   *
+ *                                                              *
+ *   http://www.apache.org/licenses/LICENSE-2.0                 *
+ *                                                              *
+ * Unless required by applicable law or agreed to in writing,   *
+ * software distributed under the License is distributed on an  *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY       *
+ * KIND, either express or implied.  See the License for the    *
+ * specific language governing permissions and limitations      *
+ * under the License.                                           *
+ ****************************************************************/
+
+package org.apache.james.mailbox.lucene.search;
+
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.ATTACHMENT_TEXT_CONTENT_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.BASE_SUBJECT_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.BCC_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.BODY_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.CC_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.FIRST_CC_MAILBOX_NAME_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.FIRST_FROM_MAILBOX_NAME_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.FIRST_TO_MAILBOX_NAME_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.FLAGS_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.FROM_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.HAS_ATTACHMENT_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.HEADERS_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.ID_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_DAY_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_HOUR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_MILLISECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_MINUTE_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_MONTH_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_SECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.INTERNAL_DATE_FIELD_YEAR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.MAILBOX_ID_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.MESSAGE_ID_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.PREFIX_HEADER_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_DAY_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_HOUR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_MINUTE_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_MONTH_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_SECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SAVE_DATE_FIELD_YEAR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_DAY_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_HOUR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_MILLISECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_MINUTE_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_MONTH_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_SECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_FIELD_YEAR_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SENT_DATE_SORT_FIELD_MILLISECOND_RESOLUTION;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SIZE_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.SUBJECT_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.THREAD_ID_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.TO_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.UID_FIELD;
+import static org.apache.james.mailbox.lucene.search.DocumentFieldConstants.USERS;
+
+import java.util.Arrays;
+import java.util.Date;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+
+import jakarta.mail.Flags;
+
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MessageUid;
+import org.apache.james.mailbox.extractor.TextExtractor;
+import org.apache.james.mailbox.model.MailboxId;
+import org.apache.james.mailbox.model.MessageAttachmentMetadata;
+import org.apache.james.mailbox.opensearch.json.HeaderCollection;
+import org.apache.james.mailbox.opensearch.json.MimePart;
+import org.apache.james.mailbox.opensearch.json.MimePartParser;
+import org.apache.james.mailbox.store.mail.model.MailboxMessage;
+import org.apache.james.mailbox.store.search.SearchUtil;
+import org.apache.lucene.document.DateTools;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.document.SortedSetDocValuesField;
+import org.apache.lucene.document.StoredField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.util.BytesRef;
+
+import com.github.fge.lambdas.Throwing;
+import com.google.common.collect.ImmutableMap;
+
+import reactor.core.publisher.Mono;
+
+public class LuceneIndexableDocument {
+    public static final Map<Flags.Flag, String> SYSTEM_FLAG_STRING_MAP = ImmutableMap.<Flags.Flag, String>builder()
+        .put(Flags.Flag.ANSWERED, "\\ANSWERED")
+        .put(Flags.Flag.DELETED, "\\DELETED")
+        .put(Flags.Flag.DRAFT, "\\DRAFT")
+        .put(Flags.Flag.FLAGGED, "\\FLAGGED")
+        .put(Flags.Flag.RECENT, "\\RECENT")
+        .put(Flags.Flag.SEEN, "\\FLAG")
+        .build();
+
+    private final TextExtractor textExtractor;
+
+    public LuceneIndexableDocument(TextExtractor textExtractor) {
+        this.textExtractor = textExtractor;
+    }
+
+    public Mono<Document> createMessageDocument(MailboxMessage message, MailboxSession session) {
+        return Throwing.supplier(() -> new MimePartParser(textExtractor).parse(message.getFullContent())).get()

Review Comment:
   If mimePartParser is needed shall we move it into `mailbox-store` rather than adding a dependency to mailbox-opensearch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827158945


##########
mailbox/lucene/src/test/java/org/apache/james/mailbox/lucene/search/LuceneFSSearchHighLighterTest.java:
##########
@@ -65,7 +66,7 @@ public void setUp(@TempDir Path tempPath) throws Exception {
             .listeningSearchIndex(Throwing.function(preInstanciationStage -> new LuceneMessageSearchIndex(
                 preInstanciationStage.getMapperFactory(), new InMemoryId.Factory(), FSDirectory.open(tempPath),
                 messageIdFactory,
-                preInstanciationStage.getSessionProvider())))
+                preInstanciationStage.getSessionProvider(), new JsoupTextExtractor())))

Review Comment:
   Cool but it will likely create Spring issues, correct?
   
   We need to fixup spring injects as well (inject JsoupTextExtractor is enough?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] WIP - JAMES-4082 Lucene search attachment [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1822132894


##########
mailbox/lucene/pom.xml:
##########
@@ -53,6 +53,10 @@
             <type>test-jar</type>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>${james.groupId}</groupId>
+            <artifactId>apache-james-mailbox-opensearch</artifactId>

Review Comment:
   No please.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827156848


##########
mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneMessageSearchIndex.java:
##########
@@ -394,23 +398,25 @@ public static class LuceneMessageSearchIndexGroup extends org.apache.james.event
 
     private final MailboxId.Factory mailboxIdFactory;
     private final MessageId.Factory messageIdFactory;
+    private final LuceneIndexableDocument indexableDocument;
 
     @VisibleForTesting
     final IndexWriter writer;
     private final Directory directory;
 
     private int maxQueryResults = DEFAULT_MAX_QUERY_RESULTS;
 
-    private boolean suffixMatch = false;
+    private boolean suffixMatch = true;

Review Comment:
   Why? What's the impact?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "vttranlina (via GitHub)" <gi...@apache.org>.
vttranlina commented on code in PR #2476:
URL: https://github.com/apache/james-project/pull/2476#discussion_r1827326835


##########
mailbox/lucene/src/main/java/org/apache/james/mailbox/lucene/search/LuceneMessageSearchIndex.java:
##########
@@ -394,23 +398,25 @@ public static class LuceneMessageSearchIndexGroup extends org.apache.james.event
 
     private final MailboxId.Factory mailboxIdFactory;
     private final MessageId.Factory messageIdFactory;
+    private final LuceneIndexableDocument indexableDocument;
 
     @VisibleForTesting
     final IndexWriter writer;
     private final Directory directory;
 
     private int maxQueryResults = DEFAULT_MAX_QUERY_RESULTS;
 
-    private boolean suffixMatch = false;
+    private boolean suffixMatch = true;

Review Comment:
   If true, it will replace `PrefixQuery` with `WildcardQuery`.
   it for `SearchCapabilities.PartialEmailMatch` 
   
   Bellow is the document related to this variable
   ```
   If set to true this implementation will use WildcardQuery to match suffix and prefix. This is what RFC3501 expects but is often not what the user does. It also slow things a lot if you have complex queries which use many "TEXT" arguments. If you want the implementation to behave strict like RFC3501 says, you should set this to true.
   The default is false for performance reasons
   ```
   
   If performance is concern, we can keep old behavior, (false)
   



##########
mailbox/lucene/src/test/java/org/apache/james/mailbox/lucene/search/LuceneMemorySearchHighLighterTest.java:
##########
@@ -112,4 +126,46 @@ public void applyRightsCommand(MailboxId mailboxId, Username owner, Username del
             MailboxACL.command().forUser(delegated).rights(MailboxACL.FULL_RIGHTS).asAddition(),
             session(owner))).run();
     }
+
+
+    @Test
+    void shouldHighlightAttachmentTextContentWhenTextBodyDoesNotMatch() throws Exception {

Review Comment:
   moved it to contract class 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4082 Adapt Lucene search - support full capability [james-project]

Posted by "vttranlina (via GitHub)" <gi...@apache.org>.
vttranlina commented on PR #2476:
URL: https://github.com/apache/james-project/pull/2476#issuecomment-2456035118

   squash 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org