You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "quantranhong1999 (via GitHub)" <gi...@apache.org> on 2024/08/14 10:30:56 UTC
[PR] JAMES-4057 Enable fuzziness search [james-project]
quantranhong1999 opened a new pull request, #2376:
URL: https://github.com/apache/james-project/pull/2376
Fuzziness search helps improve search experience as allow typos in user input. cf: https://opensearch.org/docs/latest/query-dsl/full-text/match/#fuzziness
We support TextCriterion and SubjectCriterion but not HeaderCriterion as of now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1719659152
##########
docs/modules/servers/pages/distributed/configure/opensearch.adoc:
##########
@@ -97,6 +97,11 @@ turning off headers indexing result in non-strict compliance with the IMAP / JMA
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source)
Default to false.
+| opensearch.text.fuzziness.search
+| Use fuzziness on text searches. This option helps to correct user typing mistakes.
+
+Default to true.
Review Comment:
> on the indexing and search part?
This change only affects the search phase.
> have this been performance tested
Benchmark on Email/query with random text: https://github.com/linagora/james-gatling/blob/master/src/main/scala-2.13/org/apache/james/gatling/jmap/rfc8621/JmapEmail.scala#L49
Fuzzy search off:

Fuzzy search on:

p99 94ms -> p99 173ms: indeed with fuzzy on, the OpenSearch query is slower, but it is still acceptable IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1720024074
##########
docs/modules/servers/pages/distributed/configure/opensearch.adoc:
##########
@@ -97,6 +97,11 @@ turning off headers indexing result in non-strict compliance with the IMAP / JMA
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source)
Default to false.
+| opensearch.text.fuzziness.search
+| Use fuzziness on text searches. This option helps to correct user typing mistakes.
+
+Default to true.
Review Comment:
Yes it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2376:
URL: https://github.com/apache/james-project/pull/2376#issuecomment-2293253932
> Is that related, or unstable test?
It was related. Because with fuzzy search we allow more flexible input that leads to more flexible results.
Therefore I disable fuzzy search by default, and let the admin choose himself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1721241494
##########
mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchOptimizeMoveAndFuzzySearchIntegrationTest.java:
##########
@@ -0,0 +1,91 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.opensearch;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.nio.charset.StandardCharsets;
+
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MailboxSessionUtil;
+import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.SearchQuery;
+import org.apache.james.mime4j.dom.Message;
+import org.junit.jupiter.api.Test;
+import org.opensearch.client.opensearch._types.query_dsl.QueryBuilders;
+
+import reactor.core.publisher.Flux;
+
+class OpenSearchOptimizeMoveAndFuzzySearchIntegrationTest extends OpenSearchIntegrationTest {
Review Comment:
In general running more test isn't much of an issue but OpenSearch tests are notoriously long.
What are the benefits of running the whole test suite with fuziness, all of the time?
I'd run only the test specific to fuziness in order to cut build time. We would save likely several minutes I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1721241494
##########
mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchOptimizeMoveAndFuzzySearchIntegrationTest.java:
##########
@@ -0,0 +1,91 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.opensearch;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.nio.charset.StandardCharsets;
+
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.MailboxSessionUtil;
+import org.apache.james.mailbox.MessageManager;
+import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.SearchQuery;
+import org.apache.james.mime4j.dom.Message;
+import org.junit.jupiter.api.Test;
+import org.opensearch.client.opensearch._types.query_dsl.QueryBuilders;
+
+import reactor.core.publisher.Flux;
+
+class OpenSearchOptimizeMoveAndFuzzySearchIntegrationTest extends OpenSearchIntegrationTest {
Review Comment:
In general running more test isn't much of an issue but OpenSearch tests are notoriously long.
What are the benefits of running the whole test suite with fuziness, all of the time?
I'd run only the test specific to fuziness in order to cut build time. We would save likely several minutes I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa merged PR #2376:
URL: https://github.com/apache/james-project/pull/2376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2376:
URL: https://github.com/apache/james-project/pull/2376#issuecomment-2288401695
Do you think we should support the fuzzy search for `HeaderCriterion` (e.g. address searching)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1716722048
##########
docs/modules/servers/pages/distributed/configure/opensearch.adoc:
##########
@@ -97,6 +97,11 @@ turning off headers indexing result in non-strict compliance with the IMAP / JMA
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source)
Default to false.
+| opensearch.text.fuzziness.search
+| Use fuzziness on text searches. This option helps to correct user typing mistakes.
+
+Default to true.
Review Comment:
I'm peacemealed: have this been performance tested both on the indexing and search part?
Until it is I would rather be defensive and keep this disabled by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "Arsnael (via GitHub)" <gi...@apache.org>.
Arsnael commented on PR #2376:
URL: https://github.com/apache/james-project/pull/2376#issuecomment-2290459972
https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2376/1/testReport/junit/org.apache.james.mpt.imapmailbox.elasticsearch/OpenSearchUidSearchOnIndexTest/testSearchAtomsUS/
Is that related, or unstable test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4057 Enable fuzziness search [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2376:
URL: https://github.com/apache/james-project/pull/2376#discussion_r1720018865
##########
docs/modules/servers/pages/distributed/configure/opensearch.adoc:
##########
@@ -97,6 +97,11 @@ turning off headers indexing result in non-strict compliance with the IMAP / JMA
If the message is not found, it will fall back to the old behavior (The message will be indexed from the blobStore source)
Default to false.
+| opensearch.text.fuzziness.search
+| Use fuzziness on text searches. This option helps to correct user typing mistakes.
+
+Default to true.
Review Comment:
The email query shall exercise text search for this to be relevant. Is it the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org