You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by bt...@apache.org on 2023/11/22 15:36:51 UTC

(james-project) branch master updated (3dd1b53612 -> caa28d5580)

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


    from 3dd1b53612 [UPGRADE] amqp-client 5.18.0 -> 5.20.0
     new c5d9cbf22f [BUILD] Disable JMX for MailDelivrerToHost object pool
     new 9339116155 [BUILD] Disable Metrics logger
     new 69e90ce0e4 [BUILD] Implement caching for ExtendedClassLoader
     new 3f24078c67 [BUILD] Avoid polling without delays for MemoryMailQueue
     new 7e63c3e6e7 [BUILD] Lower password hasking requirements
     new caa28d5580 [BUILD] Per test index is actually faster

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../opensearch/DockerOpenSearchExtension.java      |  4 ++-
 .../opensearch/OpenSearchIntegrationTest.java      | 31 ++++++++++++++--------
 .../src/test/resources/usersrepository.xml         |  2 +-
 .../apache/james/utils/ExtendedClassLoader.java    | 19 +++++++++++--
 .../org/apache/james/utils/GuiceGenericLoader.java |  6 +----
 .../remote/delivery/MailDelivrerToHost.java        |  1 +
 .../src/test/resources/usersrepository.xml         |  2 +-
 .../src/test/resources/usersrepository.xml         |  1 +
 .../james/queue/memory/MemoryMailQueueFactory.java |  8 +++++-
 testing/base/src/main/resources/logback-test.xml   |  3 ++-
 10 files changed, 54 insertions(+), 23 deletions(-)
 copy server/apps/{webadmin-cli => cli-integration-tests}/src/test/resources/usersrepository.xml (97%)
 copy server/{apps/webadmin-cli => protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing}/src/test/resources/usersrepository.xml (97%)


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 04/06: [BUILD] Avoid polling without delays for MemoryMailQueue

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 3f24078c67eda1124bbede01dee2a2b8e2df60f2
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Fri Nov 17 17:26:09 2023 +0100

    [BUILD] Avoid polling without delays for MemoryMailQueue
---
 .../org/apache/james/queue/memory/MemoryMailQueueFactory.java     | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java b/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
index 42770b30ae..cc5e7541c7 100644
--- a/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
+++ b/server/queue/queue-memory/src/main/java/org/apache/james/queue/memory/MemoryMailQueueFactory.java
@@ -126,7 +126,13 @@ public class MemoryMailQueueFactory implements MailQueueFactory<MemoryMailQueueF
             this.name = name;
             this.scheduler = Schedulers.newSingle("memory-mail-queue");
 
-            this.flux = Mono.<MemoryMailQueueItem>create(sink -> sink.success(mailItems.poll()))
+            this.flux = Mono.<MemoryMailQueueItem>create(sink -> {
+                    try {
+                        sink.success(mailItems.poll(10, TimeUnit.MILLISECONDS));
+                    } catch (InterruptedException e) {
+                        sink.success();
+                    }
+                })
                 .subscribeOn(Schedulers.boundedElastic())
                 .repeat()
                 .subscribeOn(scheduler)


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 05/06: [BUILD] Lower password hasking requirements

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 7e63c3e6e7f3f852c531ea7e1041d9a9bf951b52
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Fri Nov 17 17:29:56 2023 +0100

    [BUILD] Lower password hasking requirements
    
    This speed up tests
---
 .../cli-integration-tests}/src/test/resources/usersrepository.xml        | 1 +
 .../src/test/resources/usersrepository.xml                               | 1 +
 .../src/test/resources/usersrepository.xml                               | 1 +
 3 files changed, 3 insertions(+)

diff --git a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml b/server/apps/cli-integration-tests/src/test/resources/usersrepository.xml
similarity index 97%
copy from server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
copy to server/apps/cli-integration-tests/src/test/resources/usersrepository.xml
index 123ce9b71f..d7495138f6 100644
--- a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
+++ b/server/apps/cli-integration-tests/src/test/resources/usersrepository.xml
@@ -20,5 +20,6 @@
 
 <!-- See http://james.apache.org/server/3/config.html for usage -->
 <usersrepository name="LocalUsers" class="org.apache.james.user.memory.MemoryUsersRepository">
+  <algorithm>SHA-1</algorithm>
   <enableVirtualHosting>true</enableVirtualHosting>
 </usersrepository>
diff --git a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml b/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/usersrepository.xml
similarity index 97%
copy from server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
copy to server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/usersrepository.xml
index 123ce9b71f..1e9092b831 100644
--- a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
+++ b/server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/resources/usersrepository.xml
@@ -21,4 +21,5 @@
 <!-- See http://james.apache.org/server/3/config.html for usage -->
 <usersrepository name="LocalUsers" class="org.apache.james.user.memory.MemoryUsersRepository">
   <enableVirtualHosting>true</enableVirtualHosting>
+  <algorithm>SHA-1</algorithm>
 </usersrepository>
diff --git a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml b/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
index 123ce9b71f..1e9092b831 100644
--- a/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
+++ b/server/protocols/jmap-rfc-8621-integration-tests/memory-jmap-rfc-8621-integration-tests/src/test/resources/usersrepository.xml
@@ -21,4 +21,5 @@
 <!-- See http://james.apache.org/server/3/config.html for usage -->
 <usersrepository name="LocalUsers" class="org.apache.james.user.memory.MemoryUsersRepository">
   <enableVirtualHosting>true</enableVirtualHosting>
+  <algorithm>SHA-1</algorithm>
 </usersrepository>


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 02/06: [BUILD] Disable Metrics logger

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 9339116155f356e203ac25a9d60959597f3d5d96
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Fri Nov 17 13:47:26 2023 +0100

    [BUILD] Disable Metrics logger
    
    Stopping the metric logger take a few ms.
---
 testing/base/src/main/resources/logback-test.xml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testing/base/src/main/resources/logback-test.xml b/testing/base/src/main/resources/logback-test.xml
index ddcf72c376..3451bad032 100644
--- a/testing/base/src/main/resources/logback-test.xml
+++ b/testing/base/src/main/resources/logback-test.xml
@@ -28,5 +28,6 @@
                 <appender-ref ref="FILE" />
         </root>
         <logger name="org.apache.james" level="DEBUG" />
-	<logger name="org.testcontainers" level="DEBUG"/>
+        <logger name="org.testcontainers" level="DEBUG"/>
+        <logger name="org.apache.james.metrics" level="INFO"/>
 </configuration>


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 01/06: [BUILD] Disable JMX for MailDelivrerToHost object pool

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit c5d9cbf22f946eb26815b89f6880384caba85b24
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Fri Nov 17 13:46:45 2023 +0100

    [BUILD] Disable JMX for MailDelivrerToHost object pool
    
    Flame graph shows non neglictible amount of resources is wasted
     doing this.
---
 .../james/transport/mailets/remote/delivery/MailDelivrerToHost.java      | 1 +
 1 file changed, 1 insertion(+)

diff --git a/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/remote/delivery/MailDelivrerToHost.java b/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/remote/delivery/MailDelivrerToHost.java
index 5580f079cc..dc63a41a7a 100644
--- a/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/remote/delivery/MailDelivrerToHost.java
+++ b/server/mailet/mailets/src/main/java/org/apache/james/transport/mailets/remote/delivery/MailDelivrerToHost.java
@@ -84,6 +84,7 @@ public class MailDelivrerToHost {
     private ObjectPool<Session> createSessionPool(Properties defaultConfiguration) {
         GenericObjectPoolConfig<Session> poolConfig = new GenericObjectPoolConfig<>();
         poolConfig.setMaxTotal(-1); // unbounded pool, scales to match peak delivery thread concurrency
+        poolConfig.setJmxEnabled(false);
         return new GenericObjectPool<>(new BasePooledObjectFactory<>() {
             @Override
             public Session create() {


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 03/06: [BUILD] Implement caching for ExtendedClassLoader

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 69e90ce0e47a773cfb9374cca4bb03a0110560a5
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Fri Nov 17 14:29:14 2023 +0100

    [BUILD] Implement caching for ExtendedClassLoader
    
    In the tests it takes 40ms for a Memory james server to look up
    classes on the classpath.
---
 .../org/apache/james/utils/ExtendedClassLoader.java   | 19 +++++++++++++++++--
 .../org/apache/james/utils/GuiceGenericLoader.java    |  6 +-----
 2 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/server/container/guice/utils/src/main/java/org/apache/james/utils/ExtendedClassLoader.java b/server/container/guice/utils/src/main/java/org/apache/james/utils/ExtendedClassLoader.java
index 5602a68b6f..e9808c7b1d 100644
--- a/server/container/guice/utils/src/main/java/org/apache/james/utils/ExtendedClassLoader.java
+++ b/server/container/guice/utils/src/main/java/org/apache/james/utils/ExtendedClassLoader.java
@@ -24,6 +24,9 @@ import java.io.IOException;
 import java.net.MalformedURLException;
 import java.net.URL;
 import java.net.URLClassLoader;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
 import java.util.stream.Stream;
 
 import javax.inject.Inject;
@@ -36,6 +39,7 @@ import org.slf4j.LoggerFactory;
 import com.github.fge.lambdas.Throwing;
 
 public class ExtendedClassLoader {
+    private static final Map<FullyQualifiedClassName, Optional<Class<?>>> cache = new HashMap<>();
 
     private static final Logger LOGGER = LoggerFactory.getLogger(ExtendedClassLoader.class);
 
@@ -73,7 +77,18 @@ public class ExtendedClassLoader {
     }
 
     @SuppressWarnings("unchecked")
-    public <T> Class<T> locateClass(FullyQualifiedClassName className) throws ClassNotFoundException {
-        return (Class<T>) urlClassLoader.loadClass(className.getName());
+    public <T> Optional<Class<T>> locateClass(FullyQualifiedClassName className) {
+        Optional<Class<?>> cachedValue = cache.get(className);
+        if (cachedValue != null) {
+            return (Optional) cachedValue;
+        }
+        try {
+            Class<?> aClass = urlClassLoader.loadClass(className.getName());
+            cache.put(className, Optional.of(aClass));
+            return (Optional) Optional.of(aClass);
+        } catch (ClassNotFoundException e) {
+            cache.put(className, Optional.empty());
+            return Optional.empty();
+        }
     }
 }
diff --git a/server/container/guice/utils/src/main/java/org/apache/james/utils/GuiceGenericLoader.java b/server/container/guice/utils/src/main/java/org/apache/james/utils/GuiceGenericLoader.java
index f727a5cd26..997059d821 100644
--- a/server/container/guice/utils/src/main/java/org/apache/james/utils/GuiceGenericLoader.java
+++ b/server/container/guice/utils/src/main/java/org/apache/james/utils/GuiceGenericLoader.java
@@ -73,11 +73,7 @@ public class GuiceGenericLoader {
         }
 
         private Stream<Class<T>> tryLocateClass(FullyQualifiedClassName className) {
-            try {
-                return Stream.of(extendedClassLoader.locateClass(className));
-            } catch (ClassNotFoundException e) {
-                return Stream.empty();
-            }
+            return (Stream) extendedClassLoader.locateClass(className).stream();
         }
     }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


(james-project) 06/06: [BUILD] Per test index is actually faster

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit caa28d5580ffc309ad9789b7b77531a9676809dd
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Mon Nov 20 22:19:38 2023 +0100

    [BUILD] Per test index is actually faster
    
    Gains ~1 second per test
---
 .../opensearch/DockerOpenSearchExtension.java      |  4 ++-
 .../opensearch/OpenSearchIntegrationTest.java      | 31 ++++++++++++++--------
 2 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearchExtension.java b/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearchExtension.java
index 0f27bdd978..c0310c2f1a 100644
--- a/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearchExtension.java
+++ b/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearchExtension.java
@@ -19,6 +19,8 @@
 
 package org.apache.james.backends.opensearch;
 
+import java.time.Duration;
+
 import org.junit.jupiter.api.extension.AfterEachCallback;
 import org.junit.jupiter.api.extension.BeforeEachCallback;
 import org.junit.jupiter.api.extension.ExtensionContext;
@@ -32,7 +34,7 @@ import org.testcontainers.shaded.org.awaitility.Awaitility;
 public class DockerOpenSearchExtension implements AfterEachCallback, BeforeEachCallback, ParameterResolver {
 
     @FunctionalInterface
-    interface CleanupStrategy {
+    public interface CleanupStrategy {
         CleanupStrategy NONE = any -> {};
 
         void clean(DockerOpenSearch openSearch);
diff --git a/mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchIntegrationTest.java b/mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchIntegrationTest.java
index 8eece0cced..30cd8315d3 100644
--- a/mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchIntegrationTest.java
+++ b/mailbox/opensearch/src/test/java/org/apache/james/mailbox/opensearch/OpenSearchIntegrationTest.java
@@ -27,10 +27,13 @@ import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.time.ZoneId;
 import java.util.List;
+import java.util.UUID;
 
 import org.apache.james.backends.opensearch.DockerOpenSearchExtension;
+import org.apache.james.backends.opensearch.IndexName;
 import org.apache.james.backends.opensearch.OpenSearchIndexer;
 import org.apache.james.backends.opensearch.ReactorOpenSearchClient;
+import org.apache.james.backends.opensearch.ReadAliasName;
 import org.apache.james.backends.opensearch.WriteAliasName;
 import org.apache.james.mailbox.MailboxSession;
 import org.apache.james.mailbox.MailboxSessionUtil;
@@ -94,18 +97,17 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
     static TikaExtension tika = new TikaExtension();
 
     @RegisterExtension
-    static DockerOpenSearchExtension openSearch = new DockerOpenSearchExtension(
-        new DockerOpenSearchExtension.DeleteAllIndexDocumentsCleanupStrategy(new WriteAliasName("mailboxWriteAlias")));
+    static DockerOpenSearchExtension openSearch = new DockerOpenSearchExtension(DockerOpenSearchExtension.CleanupStrategy.NONE);
 
     static TikaTextExtractor textExtractor;
     static ReactorOpenSearchClient client;
+    private ReadAliasName readAliasName;
+    private WriteAliasName writeAliasName;
+    private IndexName indexName;
 
     @BeforeAll
     static void setUpAll() throws Exception {
         client = openSearch.getDockerOpenSearch().clientProvider().get();
-        MailboxIndexCreationUtil.prepareDefaultClient(
-            client,
-            openSearch.getDockerOpenSearch().configuration());
         textExtractor = new TikaTextExtractor(new RecordingMetricFactory(),
             new TikaHttpClientImpl(TikaConfiguration.builder()
                 .host(tika.getIp())
@@ -136,6 +138,13 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
 
         MailboxIdRoutingKeyFactory routingKeyFactory = new MailboxIdRoutingKeyFactory();
 
+        readAliasName = new ReadAliasName(UUID.randomUUID().toString());
+        writeAliasName = new WriteAliasName(UUID.randomUUID().toString());
+        indexName = new IndexName(UUID.randomUUID().toString());
+        MailboxIndexCreationUtil.prepareClient(
+            client, readAliasName, writeAliasName, indexName,
+            openSearch.getDockerOpenSearch().configuration());
+
         InMemoryIntegrationResources resources = InMemoryIntegrationResources.builder()
             .preProvisionnedFakeAuthenticator()
             .fakeAuthorizator()
@@ -146,9 +155,9 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
                 preInstanciationStage.getMapperFactory(),
                 ImmutableSet.of(),
                 new OpenSearchIndexer(client,
-                    MailboxOpenSearchConstants.DEFAULT_MAILBOX_WRITE_ALIAS),
+                    writeAliasName),
                 new OpenSearchSearcher(client, new QueryConverter(new CriterionConverter()), SEARCH_SIZE,
-                    MailboxOpenSearchConstants.DEFAULT_MAILBOX_READ_ALIAS, routingKeyFactory),
+                    readAliasName, routingKeyFactory),
                 new MessageToOpenSearchJson(textExtractor, ZoneId.of("Europe/Paris"), IndexAttachments.YES, IndexHeaders.YES),
                 preInstanciationStage.getSessionProvider(), routingKeyFactory, messageIdFactory,
                 openSearchMailboxConfiguration(), new RecordingMetricFactory()))
@@ -201,7 +210,7 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
 
         ObjectNode updatedDocument = client.get(
                  new GetRequest.Builder()
-                    .index("mailboxWriteAlias")
+                    .index(indexName.getValue())
                     .id(mailboxBId.serialize() + ":" + bMessageUid.asLong())
                     .routing(mailboxBId.serialize())
                     .build())
@@ -235,7 +244,7 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
 
         // Try to delete the document manually to simulate a not found document.
         client.deleteByQuery(new DeleteByQueryRequest.Builder()
-                .index("mailboxWriteAlias")
+                .index(indexName.getValue())
                 .query(new MatchAllQuery.Builder().build()._toQuery())
                 .build())
             .block();
@@ -283,7 +292,7 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
         CALMLY_AWAIT.atMost(Durations.TEN_SECONDS)
             .untilAsserted(() -> assertThat(client.search(
                     new SearchRequest.Builder()
-                        .index(MailboxOpenSearchConstants.DEFAULT_MAILBOX_INDEX.getValue())
+                        .index(indexName.getValue())
                         .query(QueryBuilders.matchAll().build()._toQuery())
                         .build())
                 .block()
@@ -564,7 +573,7 @@ class OpenSearchIntegrationTest extends AbstractMessageSearchIndexTest {
         CALMLY_AWAIT.atMost(Durations.TEN_SECONDS)
                 .untilAsserted(() -> assertThat(client.search(
                         new SearchRequest.Builder()
-                            .index(MailboxOpenSearchConstants.DEFAULT_MAILBOX_INDEX.getValue())
+                            .index(indexName.getValue())
                             .query(query)
                             .build())
                         .block()


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org