You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by bt...@apache.org on 2021/12/01 02:51:34 UTC

[james-project] branch 3.6.x updated (3926a3d -> 23c496e)

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a change to branch 3.6.x
in repository https://gitbox.apache.org/repos/asf/james-project.git.


    from 3926a3d  JAMES-1436 SwitchableLineBasedFrameDecoder: clean up cumulation buffer (#674)
     new 8fd9fc0  JAMES-3676 S3 blob store should not fail upon many partial reads
     new a74a8e5  JAMES-3676 ReactorUtils.toInputStream should cancel publisher subscriptions upon partial reads
     new 23c496e  JAMES-3676 Store should close related streams...

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/james/blob/api/Store.java |  4 +--
 .../blob/objectstorage/aws/S3BlobStoreDAOTest.java | 33 ++++++++++++++++++++++
 .../java/org/apache/james/util/ReactorUtils.java   | 13 +++++++--
 .../org/apache/james/util/ReactorUtilsTest.java    | 19 +++++++++++++
 4 files changed, 64 insertions(+), 5 deletions(-)

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


[james-project] 01/03: JAMES-3676 S3 blob store should not fail upon many partial reads

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch 3.6.x
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 8fd9fc087954143b28fd540bacd2e239b03a41c5
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Nov 25 14:20:27 2021 +0700

    JAMES-3676 S3 blob store should not fail upon many partial reads
---
 .../blob/objectstorage/aws/S3BlobStoreDAOTest.java | 33 ++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAOTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAOTest.java
index e426926..80e8e6a 100644
--- a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAOTest.java
+++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3BlobStoreDAOTest.java
@@ -18,13 +18,27 @@
  ****************************************************************/
 package org.apache.james.blob.objectstorage.aws;
 
+import static org.apache.james.blob.api.BlobStoreDAOFixture.ELEVEN_KILOBYTES;
+import static org.apache.james.blob.api.BlobStoreDAOFixture.TEST_BUCKET_NAME;
+import static org.assertj.core.api.Assertions.assertThatCode;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.stream.IntStream;
+
 import org.apache.james.blob.api.BlobStoreDAO;
 import org.apache.james.blob.api.BlobStoreDAOContract;
+import org.apache.james.blob.api.TestBlobId;
 import org.junit.jupiter.api.AfterAll;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 
+import com.google.common.io.ByteSource;
+
+import reactor.core.publisher.Mono;
+
 @ExtendWith(DockerAwsS3Extension.class)
 public class S3BlobStoreDAOTest implements BlobStoreDAOContract {
     private static S3BlobStoreDAO testee;
@@ -59,4 +73,23 @@ public class S3BlobStoreDAOTest implements BlobStoreDAOContract {
     public BlobStoreDAO testee() {
         return testee;
     }
+
+    @Test
+    void readShouldNotLeakHttpConnexionsForUnclosedStreams() {
+        BlobStoreDAO store = testee();
+
+        TestBlobId blobId = new TestBlobId("id");
+        Mono.from(store.save(TEST_BUCKET_NAME, blobId, ByteSource.wrap(ELEVEN_KILOBYTES))).block();
+
+        assertThatCode(() -> IntStream.range(0, 256)
+            .forEach(i -> {
+                InputStream inputStream = store.read(TEST_BUCKET_NAME, blobId);
+                // Close the stream without reading it
+                try {
+                    inputStream.close();
+                } catch (IOException e) {
+                    throw new RuntimeException(e);
+                }
+            })).doesNotThrowAnyException();
+    }
 }

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


[james-project] 02/03: JAMES-3676 ReactorUtils.toInputStream should cancel publisher subscriptions upon partial reads

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch 3.6.x
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit a74a8e59dae0d2c9283b719d87cfe589c81036cf
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Nov 25 14:21:16 2021 +0700

    JAMES-3676 ReactorUtils.toInputStream should cancel publisher subscriptions upon partial reads
---
 .../main/java/org/apache/james/util/ReactorUtils.java | 13 ++++++++++---
 .../java/org/apache/james/util/ReactorUtilsTest.java  | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java b/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
index c11a9d4..f678118 100644
--- a/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
+++ b/server/container/util/src/main/java/org/apache/james/util/ReactorUtils.java
@@ -28,6 +28,7 @@ import java.util.Optional;
 import java.util.function.BiConsumer;
 import java.util.function.Consumer;
 import java.util.function.Function;
+import java.util.stream.Stream;
 
 import org.reactivestreams.Publisher;
 import org.slf4j.Logger;
@@ -89,7 +90,7 @@ public class ReactorUtils {
     }
 
     public static InputStream toInputStream(Flux<ByteBuffer> byteArrays) {
-        return new StreamInputStream(byteArrays.toIterable(1).iterator());
+        return new StreamInputStream(byteArrays.toStream(1));
     }
 
     public static Flux<ByteBuffer> toChunks(InputStream inputStream, int bufferSize) {
@@ -112,10 +113,12 @@ public class ReactorUtils {
         private static final int NO_MORE_DATA = -1;
 
         private final Iterator<ByteBuffer> source;
+        private final Stream<ByteBuffer> sourceAsStream;
         private Optional<ByteBuffer> currentItemByteStream;
 
-        StreamInputStream(Iterator<ByteBuffer> source) {
-            this.source = source;
+        StreamInputStream(Stream<ByteBuffer> source) {
+            this.source = source.iterator();
+            this.sourceAsStream = source;
             this.currentItemByteStream = Optional.empty();
         }
 
@@ -150,6 +153,10 @@ public class ReactorUtils {
             return currentItemByteStream;
         }
 
+        @Override
+        public void close() throws IOException {
+            sourceAsStream.close();
+        }
     }
 
     private static int byteToInt(ByteBuffer buffer) {
diff --git a/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java b/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
index 448b816..9e61d3c 100644
--- a/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
+++ b/server/container/util/src/test/java/org/apache/james/util/ReactorUtilsTest.java
@@ -32,6 +32,7 @@ import java.time.Duration;
 import java.util.List;
 import java.util.concurrent.ConcurrentLinkedDeque;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
 
@@ -476,6 +477,24 @@ class ReactorUtilsTest {
         }
 
         @Test
+        void givenALongFluxBytesWhenIReadItPartiallyBeforeClosingItThenTheOriginalFluxShouldBeDisposed() throws Exception {
+            byte[] bytes = RandomStringUtils.randomAlphabetic(41111).getBytes(StandardCharsets.US_ASCII);
+
+            AtomicBoolean canceled = new AtomicBoolean(false);
+            Flux<ByteBuffer> source = Flux.fromIterable(Bytes.asList(bytes))
+                .window(3)
+                .flatMapSequential(Flux::collectList)
+                .map(Bytes::toArray)
+                .map(ByteBuffer::wrap)
+                .doOnCancel(() -> canceled.set(true));
+
+            InputStream inputStream = ReactorUtils.toInputStream(source);
+            inputStream.close();
+
+            assertThat(canceled.get()).isTrue();
+        }
+
+        @Test
         void givenAFluxOnOneByteShouldConsumeOnlyTheReadBytesAndThePrefetch() throws IOException, InterruptedException {
             AtomicInteger generateElements = new AtomicInteger(0);
             Flux<ByteBuffer> source = Flux.range(0, 10)

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


[james-project] 03/03: JAMES-3676 Store should close related streams...

Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch 3.6.x
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 23c496eaa46a70a375ed8f4059d6ff7e74bcbd35
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu Nov 25 14:33:47 2021 +0700

    JAMES-3676 Store should close related streams...
---
 .../blob-common/src/main/java/org/apache/james/blob/api/Store.java    | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/server/blob/blob-common/src/main/java/org/apache/james/blob/api/Store.java b/server/blob/blob-common/src/main/java/org/apache/james/blob/api/Store.java
index 385719c..f987fef 100644
--- a/server/blob/blob-common/src/main/java/org/apache/james/blob/api/Store.java
+++ b/server/blob/blob-common/src/main/java/org/apache/james/blob/api/Store.java
@@ -110,8 +110,8 @@ public interface Store<T, I> {
 
         private CloseableByteSource readByteSource(BucketName bucketName, BlobId blobId, StoragePolicy storagePolicy) {
             FileBackedOutputStream out = new FileBackedOutputStream(FILE_THRESHOLD);
-            try {
-                blobStore.read(bucketName, blobId, storagePolicy).transferTo(out);
+            try (InputStream in = blobStore.read(bucketName, blobId, storagePolicy)) {
+                in.transferTo(out);
                 return new DelegateCloseableByteSource(out.asByteSource(), out::reset);
             } catch (IOException e) {
                 throw new RuntimeException(e);

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org