You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/08 12:50:28 UTC

[GitHub] [druid] cryptoe opened a new pull request, #12874: Introducing Storage connector Inteface

cryptoe opened a new pull request, #12874:
URL: https://github.com/apache/druid/pull/12874

   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
   
   ### Description
   
   In the current druid code base, we have the interface `DataSegmentPusher` which allows us to push segments to the appropriate deep storage without the extension being worried about the semantics of how to push too deep storage. 
   
   While working on #12262, whose some part of the code will go as an extension, I realized that we do not have an interface that allows us to do basic "write, get, delete, deleteAll" operations on the appropriate deep storage without let's say pulling the `s3-storage-extension` dependency in the custom extension. 
   
   
   Hence, the idea of `StorageConnector` was born where the storage connector sits inside the druid core so all extensions have access to it. 
   
   Each deep storage implementation, for eg s3, GCS, will implement this interface. 
   Nowow with some jackson magic, we bind the implementation of the correct deep storage implementation on runtime using a type variable. 
   
   From the javadocs: 
   ```
   Low level interface for interacting with different storage providers like S3, GCS, Azure, local file system.
   For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
   For using the interface in your extension as a consumer, used JsonConfigProvider like:
    1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
    // bind the storage config provider
    2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
    // Used Named annotations to access the storageConnector instance in your custom extension.
    3. @Custom StorageConnector storageConnector
   ```
   
   In the future, we should clean up the `DataSegmentPuller, DataSementPusher` implementations to also use this low level storage connector.
   
   
   As part of the PR, S3StorageConnector implementation is provided which uses the s3 multi-part upload apis.
   
   
   
   
   
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `StorageConnector`
    * `S3StorageConnector`
    * `RetriableS3OutputStream`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943772325


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");

Review Comment:
   I added the documentation as part of the validation method. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943732519


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",

Review Comment:
   Yeah the user of this class should not be worried about the chunksize. 
   Move the logic to s3OutputConfig for validation there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943761483


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the path the underlying storage system. Most implementations prepend the input path
+   * with a basePath.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the path in the underlying storage system.
+   * Most implementations prepend the input path with a basePath.
+   * Callers are adivised to namespace there files as there might be race conditions.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;

Review Comment:
   Yup, that was my thought process. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on PR #12874:
URL: https://github.com/apache/druid/pull/12874#issuecomment-1210744647

   > > In the future, we should clean up the DataSegmentPuller, DataSementPusher implementations to also use this low-level storage connector.
   > 
   > Does this mean all current deep storage implementations including GCS, Azure, Alibaba OSS need to change to use these new API?
   
   Eventually, I guess. I think once we have marked the interface stable, we can slowly start migrating a few of the core data segment pusher and puller apis to talk with the storage connector. 
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942571029


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch

Review Comment:
   Thanks for catching this. Before this I was not using the java doc viewer so aggresively but from now on I will try to use that feature more :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942535018


##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();
+    String uuid2 = uuid_base + "/" + UUID.randomUUID();
+
+    createFile(uuid1);
+    createFile(uuid2);
+
+    Assert.assertTrue(storageConnector.pathExists(uuid1));
+    Assert.assertTrue(storageConnector.pathExists(uuid2));
+
+    checkContents(uuid1);
+    checkContents(uuid2);
+
+    File baseFile = new File(tempDir.getAbsolutePath() + "/" + uuid_base);
+    Assert.assertTrue(baseFile.exists());
+    Assert.assertTrue(baseFile.isDirectory());
+    Assert.assertEquals(2, baseFile.listFiles().length);
+
+    storageConnector.deleteRecursively(uuid_base);
+    Assert.assertFalse(baseFile.exists());
+
+  }
+
+  private void checkContents(String uuid) throws IOException
+  {
+    InputStream inputStream = storageConnector.read(uuid);
+    Assert.assertEquals(1, inputStream.read());
+    Assert.assertEquals(0, inputStream.available());
+    inputStream.close();
+  }
+
+  private void createFile(String uuid) throws IOException

Review Comment:
   Sure!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942537776


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   yes cause s3Connector.delete* do not throw the IO exceptions. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942537776


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   yes cause s3Connector.delete* do not throw the exceptions. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r941387110


##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector
+{
+
+  public final String basePath;

Review Comment:
   Good catch. Changed it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12874:
URL: https://github.com/apache/druid/pull/12874#issuecomment-1209484992

   > In the future, we should clean up the DataSegmentPuller, DataSementPusher implementations to also use this low-level storage connector.
   
   Does this mean all current deep storage implementations including GCS, Azure, Alibaba OSS need to change to use these new API?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942547953


##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();

Review Comment:
   As I need the string further down for the assert, I will keep it as a string for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #12874: Introducing Storage connector Inteface

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r940213106


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.

Review Comment:
   what is going to consume this interface? 



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.ISE;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty

Review Comment:
   nit - maybe explicitly declare the property name? 



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector

Review Comment:
   can `Local*` classes be moved to `local` package? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.

Review Comment:
   ```suggestion
    * The upload can be retried when it fails with transient errors.
   ```



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.ISE;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new ISE(

Review Comment:
   IMO this should be a `RE` and not `ISE` since we don't know enough about this `IOException`. 



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.

Review Comment:
   what is the expectation if the path represents a directory? 



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException

Review Comment:
   you should also add one other folder here in basePath and verify that it doesn't get deleted. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] rohangarg commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r944118447


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3StorageConnectorModule.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.inject.Binder;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.Collections;
+import java.util.List;
+
+public class S3StorageConnectorModule implements DruidModule
+{
+  @Override
+  public List<? extends Module> getJacksonModules()
+  {
+    return Collections.singletonList(
+        new SimpleModule(this.getClass().getSimpleName()).registerSubtypes(S3StorageConnectorProvider.class)
+    );
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+  }

Review Comment:
   I haven't tried it but I think `PolyBind` contains methods for doing this. Every storage connector can bind to a `PolyBind.optionalBinder` with the scheme it supports.
   And the consumer can then pull the the storage connector it needs via `PolyBind.createChoice` where the consumer property which generate the choice.
   An example of this happens with MetricsFactory injection, if that's analogous to this usecase.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943734184


##########
extensions-core/s3-extensions/pom.xml:
##########
@@ -116,6 +116,11 @@
       <artifactId>aws-java-sdk-sts</artifactId>
       <version>${aws.sdk.version}</version>
     </dependency>
+    <dependency>
+      <groupId>it.unimi.dsi</groupId>
+      <artifactId>fastutil-core</artifactId>
+      <scope>provided</scope>

Review Comment:
   Yeah Druid processing already has this jar. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943683939


##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetryableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetryableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100_000_000_000L;
+    chunkSize = 9000_000L;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[9000000] is too small for maxResultsSize[100000000000]. chunkSize should be at least [10000000]"
+    );
+    new RetryableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooSmallChunkSizeMaxResultsSizeIsNotRetionalToMaxPartNum() throws IOException
+  {
+    maxResultsSize = 274_877_906_944L;
+    chunkSize = 2_7487_790;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[27487790] is too small for maxResultsSize[274877906944]. chunkSize should be at least [27487791]"
+    );
+    new RetryableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooLargeChunkSize() throws IOException
+  {
+    maxResultsSize = 1024L * 1024 * 1024 * 1024;
+    chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES + 1;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[5368709121] should be smaller than [5368709120]"
+    );
+    new RetryableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testWriteAndHappy() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testWriteSizeLargerThanConfiguredMaxChunkSizeShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES * 3);
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      bb.clear();
+      bb.putInt(1);
+      bb.putInt(2);
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+    // each chunk 10 bytes, so there should be 2 chunks.
+    Assert.assertEquals(2, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 3);
+  }
+
+  @Test
+  public void testWriteSmallBufferShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 128;
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 600; i++) {
+        out.write(i);
+      }
+    }
+    // each chunk 128 bytes, so there should be 5 chunks.
+    Assert.assertEquals(5, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, 600);
+  }
+
+  @Test
+  public void testHitResultsSizeLimit() throws IOException
+  {
+    maxResultsSize = 50;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 14; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      Assert.assertThrows(
+          "Exceeded max results size [50]",
+          IOException.class,
+          () -> {
+            bb.clear();
+            bb.putInt(14);
+            out.write(bb.array());
+          }
+      );
+    }
+
+    s3.assertAborted();
+  }
+
+  @Test
+  public void testSuccessToUploadAfterRetry() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(1);
+
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testFailToUploadAfterRetries() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(3);
+
+    maxResultsSize = 1000;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetryableS3OutputStream out = new RetryableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 2; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      expectedException.expect(RuntimeException.class);
+      expectedException.expectCause(CoreMatchers.instanceOf(AmazonClientException.class));
+      expectedException.expectMessage("Upload failure test. Remaining failures [1]");
+      bb.clear();
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+
+    s3.assertAborted();
+  }
+
+  private static class TestAmazonS3 extends ServerSideEncryptingAmazonS3
+  {
+    private final List<UploadPartRequest> partRequests = new ArrayList<>();
+
+    private int uploadFailuresLeft;
+    private boolean aborted = false;
+    @Nullable
+    private CompleteMultipartUploadRequest completeRequest;
+
+    private TestAmazonS3(int totalUploadFailures)
+    {
+      super(EasyMock.createMock(AmazonS3.class), new NoopServerSideEncryption());
+      this.uploadFailuresLeft = totalUploadFailures;
+    }
+
+    @Override
+    public InitiateMultipartUploadResult initiateMultipartUpload(InitiateMultipartUploadRequest request)
+        throws SdkClientException
+    {
+      InitiateMultipartUploadResult result = new InitiateMultipartUploadResult();
+      result.setUploadId("uploadId");
+      return result;
+    }
+
+    @Override
+    public UploadPartResult uploadPart(UploadPartRequest request) throws SdkClientException
+    {
+      if (uploadFailuresLeft > 0) {
+        throw new AmazonClientException(
+            new IOE("Upload failure test. Remaining failures [%s]", --uploadFailuresLeft)
+        );
+      }
+      partRequests.add(request);
+      UploadPartResult result = new UploadPartResult();
+      result.setETag(StringUtils.format("%s", request.getPartNumber()));
+      result.setPartNumber(request.getPartNumber());
+      return result;
+    }
+
+    @Override
+    public void abortMultipartUpload(AbortMultipartUploadRequest request) throws SdkClientException
+    {
+      aborted = true;

Review Comment:
   Due to the sensitivity of this term in some countries, can we perhaps use `canceled` instead?



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the path the underlying storage system. Most implementations prepend the input path
+   * with a basePath.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the path in the underlying storage system.
+   * Most implementations prepend the input path with a basePath.
+   * Callers are adivised to namespace there files as there might be race conditions.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;

Review Comment:
   What are the semantics on a crash? Is the file left half written? Is it the responsibility of the caller to clean up such "orphan" files later?
   
   What are the semantics for replace? Do we support an atomic replace? Or, is that something that the caller must provide?
   
   I assume that, since this is a low-level API, the fancy stuff would be added by another layer on top. Correct?



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.

Review Comment:
   Thanks much for providing Javadoc! It is tedious to write, but is _so_ useful to consumers of the API!
   
   This comment:
   
   > Most implementations prepend the input path with a basePath.
   
   Could be in the class Javadoc:
   
   > Most implementations provide a base path within the target file system. For example, for a local file system, the base path might be /tmp/druid. Each path given in this API is relative to that base path. The implementation computed the full path from the bast path and relative path provided in the call.
   
   Then, the info won't have to be repeated for every API method.



##########
core/src/main/java/org/apache/druid/storage/local/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.storage.StorageConnector;
+import org.apache.druid.storage.StorageConnectorProvider;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new RE(

Review Comment:
   `throw IAE` since the error is presumably related to the base path argument?



##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createAndPopulateFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());

Review Comment:
   ```java
   Assert.assertTrue(new File(tempDir, uuid).exists());
   ```
   
   Here and below.



##########
extensions-core/s3-extensions/pom.xml:
##########
@@ -116,6 +116,11 @@
       <artifactId>aws-java-sdk-sts</artifactId>
       <version>${aws.sdk.version}</version>
     </dependency>
+    <dependency>
+      <groupId>it.unimi.dsi</groupId>
+      <artifactId>fastutil-core</artifactId>
+      <scope>provided</scope>

Review Comment:
   Provided means that, at run time, in production, this jar will already be on the class path. Is that true in this case? Does core Druid or S3 already include this jar? If not, the scope has to be the default so that the jar is downloaded and put on the class path.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3StorageConnectorModule.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.inject.Binder;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.Collections;
+import java.util.List;
+
+public class S3StorageConnectorModule implements DruidModule
+{
+  @Override
+  public List<? extends Module> getJacksonModules()
+  {
+    return Collections.singletonList(
+        new SimpleModule(this.getClass().getSimpleName()).registerSubtypes(S3StorageConnectorProvider.class)
+    );
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+  }

Review Comment:
   Thanks for the explanation. That hint about namespace based singleton clears up the confusion.



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   Interfaces should not need to suppress warnings. The implementation can omit the throw clause:
   
   Interface:
   ```java
   void deleteFile(String path) throws IOException;
   ```
   
   S3 implementation:
   
   ```java
   public void deleteFile(String path) ...
   ```
   
   No `@SuppressWarnings` needed.



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>

Review Comment:
   This example explains a lot! Thanks for providing it.
   
   Just a formatting nit:
   
   ```text
   <code><pre>
   JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
   // bind the storage config provider
   binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class)
   // Use Named annotations to access the storageConnector instance in your custom extension.
   @Custom StorageConnector storageConnector
   </pre></code>
   ```
   
   Maybe add a note about:
   
   > configure the storage provider in the runtime.properties file:
   >
   > druid.extension.custom.type="s3"
   > druid.extension.custom.bucket="myBucket"
   > ...
   
   Also, "druid,extension..." should be "druid.extension" (dot rather than comma).
   



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")

Review Comment:
   See comment about `@SuppressWarnings` below.



##########
core/src/main/java/org/apache/druid/storage/local/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.storage.StorageConnector;
+import org.apache.druid.storage.StorageConnectorProvider;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new RE(
+          e,
+          "Unable to create storage connector[%s] for basepath[%s]",

Review Comment:
   Suggestion:
   
   ```
   "Unable to create %s for base path [%s]"
   ```
   
   Desired result:
   
   ```
   Unable to create LocalFileStorageConnectorProvider for base path [~bob]
   ```



##########
core/src/main/java/org/apache/druid/storage/local/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.StorageConnector;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+/**
+ * Implementation that uses local filesystem. All paths are appended with the base path, in such a way that its not visible
+ * to the users of this class.
+ */
+public class LocalFileStorageConnector implements StorageConnector
+{
+
+  private final String basePath;
+
+  public LocalFileStorageConnector(String basePath) throws IOException
+  {
+    this.basePath = basePath;
+    FileUtils.mkdirp(new File(basePath));
+  }
+
+  @Override
+  public boolean pathExists(String path)
+  {
+    return fileWithBasePath(path).exists();
+  }
+
+  /**
+   * Reads the file present as basePath + path. Will throw an IO exception in case the file is not present.
+   * Closing of the stream is the responsibility of the caller.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  @Override
+  public InputStream read(String path) throws IOException
+  {
+    return Files.newInputStream(fileWithBasePath(path).toPath());
+  }
+
+  /**
+   * Writes the file present with the materialized location as basePath + path.
+   * In case the parent directory does not exist, we create the parent dir recursively.
+   * Closing of the stream is the responsibility of the caller.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  @Override
+  public OutputStream write(String path) throws IOException
+  {
+    File toWrite = fileWithBasePath(path);
+    FileUtils.mkdirp(toWrite.getParentFile());
+    return Files.newOutputStream(toWrite.toPath());
+  }
+
+  /**
+   * Deletes the file present at the location basePath + path. Throws an excecption in case a dir is encountered.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @Override
+  public void deleteFile(String path) throws IOException
+  {
+    File toDelete = fileWithBasePath(path);
+    if (toDelete.isDirectory()) {
+      throw new IAE(StringUtils.format(
+          "Found a directory on path[%s]. Please use deleteRecusively to delete dirs", path));
+    }
+    Files.delete(fileWithBasePath(path).toPath());
+  }
+
+  /**
+   * Deletes the files and sub dirs present at the basePath + dirName. Also removes the dirName
+   *
+   * @param dirName path
+   * @throws IOException
+   */
+  @Override
+  public void deleteRecursively(String dirName) throws IOException
+  {
+    FileUtils.deleteDirectory(fileWithBasePath(dirName));
+  }
+
+  private File fileWithBasePath(String path)
+  {
+    return new File(Paths.get(basePath, path).toString());

Review Comment:
   This works, but the following is simpler:
   
   ```java
   return new File(new File(basePath), path);
   ```
   
   In fact, we could just store `basePath` as a `File`, then we just need:
   
   ```java
   return new File(basePath, path);
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;

Review Comment:
   Since the API here is generic, clients would find themselves needing to be aware of the difference between local, S3, GCP, etc. A good future enhancement is that clients just write data and the implementations handle any chunking. Maybe leave an issue for that also: the issue will become clear as we use implementations other than S3.



##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createAndPopulateFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.deleteFile(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String topLeveldir = "top" + UUID.randomUUID();
+    String uuid_base = topLeveldir + "/" + UUID.randomUUID();

Review Comment:
   Nit: `topLevelDir` - consistent camel casing.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;

Review Comment:
   Thanks for filing the issue.



##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();

Review Comment:
   Would be great to use JUnit temporary files here. Is there a reason why they won't work? A bit of a pain for the reader to have to work out how temp dirs work in this test and why they have to be unlike other tests that use the JUnit mechanism.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");

Review Comment:
   Since this is an S3 thing, can we link to the S3 docs so the reader know where to find the info?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;

Review Comment:
   Over time, we should have a unified config system so that there is a single Druid temp dir, and each extension carves out a subdirectory within it. Else, it is up to the poor user to copy/paste the same path into multiple places in the configs. To us developers, each extension is its own thing. To the user, there is one Druid and one file system, they don't want to have to set a half dozen different "temp" directories.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java:
##########
@@ -43,9 +51,10 @@
 
 /**
  * {@link AmazonS3} wrapper with {@link ServerSideEncryption}. Every {@link AmazonS3#putObject},
- * {@link AmazonS3#copyObject}, {@link AmazonS3#getObject}, and {@link AmazonS3#getObjectMetadata} methods should be
+ * {@link AmazonS3#copyObject}, {@link AmazonS3#getObject}, and {@link AmazonS3#getObjectMetadata},
+ * {@link AmazonS3#initiateMultipartUpload}, @{@link AmazonS3#uploadPart} methods should be
  * wrapped using ServerSideEncryption.
- *
+ * <p>

Review Comment:
   Note for future work: if encryption is needed, and there is only one right way to do it, then it should be done automatically by an `EncryptedS3StorageConnector` rather than forcing clients to add the code. Forcing clients to do that means clients have to be aware of the target file system, which is what this great new API tries to avoid.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;

Review Comment:
   Is putting credentials in the config file the correct solution? Should there be some form of credentials provider? In this day and age, folks usually don't put their actual S3 credentials in files that are presumably put in Git for configuration-as-code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943791649


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.

Review Comment:
   As the above class level description is already too long, I think having the documentation per API makes the caller less likely to run into issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943789735


##########
core/src/main/java/org/apache/druid/storage/local/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.storage.StorageConnector;
+import org.apache.druid.storage.StorageConnectorProvider;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new RE(

Review Comment:
   Yeah IAE seems better. 
   Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943810311


##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  @Rule
+  TemporaryFolder temporaryFolder = new TemporaryFolder();
+  private File tempDir = temporaryFolder.newFolder();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir).get();
+
+  public LocalFileStorageConnectorTest() throws IOException {}

Review Comment:
   as tempFolder.newFolder() throws an exception, the constructor is marked with this but I do see that our checkstyle would most likely fail. 
   Adjusted the code to make it pass.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942549055


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.

Review Comment:
   Changed the signature and the javadocs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] LakshSingla commented on a diff in pull request #12874: Introducing Storage connector Inteface

Posted by GitBox <gi...@apache.org>.
LakshSingla commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r940955912


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream

Review Comment:
   This class seems to be solving a wider problem of chunking the multipart upload. Can a follow up to this class be that it is generic (chunking part), and then depending on the cloud storage that we are using, their constructor/specific methods can handle the upload part? Or is this something which is specific enough to S3 that this refactoring is not required?



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();
+    String uuid2 = uuid_base + "/" + UUID.randomUUID();
+
+    createFile(uuid1);
+    createFile(uuid2);
+
+    Assert.assertTrue(storageConnector.pathExists(uuid1));
+    Assert.assertTrue(storageConnector.pathExists(uuid2));
+
+    checkContents(uuid1);
+    checkContents(uuid2);
+
+    File baseFile = new File(tempDir.getAbsolutePath() + "/" + uuid_base);
+    Assert.assertTrue(baseFile.exists());
+    Assert.assertTrue(baseFile.isDirectory());
+    Assert.assertEquals(2, baseFile.listFiles().length);
+
+    storageConnector.deleteRecursively(uuid_base);
+    Assert.assertFalse(baseFile.exists());
+
+  }
+
+  private void checkContents(String uuid) throws IOException
+  {
+    InputStream inputStream = storageConnector.read(uuid);
+    Assert.assertEquals(1, inputStream.read());
+    Assert.assertEquals(0, inputStream.available());
+    inputStream.close();
+  }
+
+  private void createFile(String uuid) throws IOException

Review Comment:
   Since this method is populating the file with some data as well, should this be called something like `createAndPopulateFile`?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.

Review Comment:
   Shouldn't the chunk size decrease in compression? 



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   Is `@SuppressWarnings` required here? 



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();

Review Comment:
   I think `Paths.get()` or some other API that handles the filename nuances might be cleaner than manually joining the path.



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector

Review Comment:
   Can you please add Javadocs for this class and its methods, and also the behavior of methods like `read` and `write` on already existing files, or paths that do not exist? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943731709


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");
+
+  /**
+   * Max number of tries for each upload.
+   */
+  @JsonProperty
+  private int maxTriesOnTransientErrors = RetryUtils.DEFAULT_MAX_TRIES;
+
+  public String getBucket()
+  {
+    return bucket;
+  }
+
+  public String getPrefix()
+  {
+    return prefix;
+  }
+
+  public File getTempDir()
+  {
+    return tempDir;
+  }
+
+  @Nullable
+  public Long getChunkSize()
+  {
+    return chunkSize == null ? null : chunkSize.getBytes();
+  }
+
+  public long getMaxResultsSize()
+  {
+    return maxResultsSize.getBytes();
+  }
+
+  public int getMaxTriesOnTransientError()

Review Comment:
   updated. This seems much better



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 merged pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 merged PR #12874:
URL: https://github.com/apache/druid/pull/12874


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942479194


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;

Review Comment:
   That would be really nice to have but I think we are not there yet. We would have to change a lot of places to get there. 
   I will file a GitHub issue for this. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");

Review Comment:
   I updated the documentation to object . They need not be query results but any kind of objects. 
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId

Review Comment:
   Ahh this is not longer needed now. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");
+
+  /**
+   * Max number of tries for each upload.
+   */
+  @JsonProperty
+  private int maxTriesOnTransientErrors = RetryUtils.DEFAULT_MAX_TRIES;
+
+  public String getBucket()
+  {
+    return bucket;
+  }
+
+  public String getPrefix()
+  {
+    return prefix;
+  }
+
+  public File getTempDir()
+  {
+    return tempDir;
+  }
+
+  @Nullable
+  public Long getChunkSize()
+  {
+    return chunkSize == null ? null : chunkSize.getBytes();

Review Comment:
   Move the logic in this class. That seemed better



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;
+    chunkSize = 9000000L;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[9000000] is too small for maxResultsSize[100000000000]. chunkSize should be at least [10000000]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooSmallChunkSizeMaxResultsSizeIsNotRetionalToMaxPartNum() throws IOException
+  {
+    maxResultsSize = 274877906944L;
+    chunkSize = 27487790;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[27487790] is too small for maxResultsSize[274877906944]. chunkSize should be at least [27487791]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooLargeChunkSize() throws IOException
+  {
+    maxResultsSize = 1024L * 1024 * 1024 * 1024;
+    chunkSize = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE + 1;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[5368709121] should be smaller than [5368709120]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testWriteAndHappy() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testWriteSizeLargerThanConfiguredMaxChunkSizeShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES * 3);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      bb.clear();
+      bb.putInt(1);
+      bb.putInt(2);
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+    // each chunk 10 bytes, so there should be 2 chunks.
+    Assert.assertEquals(2, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 3);
+  }
+
+  @Test
+  public void testWriteSmallBufferShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 128;
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 600; i++) {
+        out.write(i);
+      }
+    }
+    // each chunk 128 bytes, so there should be 5 chunks.
+    Assert.assertEquals(5, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, 600);
+  }
+
+  @Test
+  public void testHitResultsSizeLimit() throws IOException
+  {
+    maxResultsSize = 50;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(

Review Comment:
   Arent we doing that now?
   As this test case specifically checks the chunking logic for the Retryable OutputStream hence not going throught the connector.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream

Review Comment:
   I think for now this is quite tied to s3 due to the way the multipart api works. 
   The decision to refactor the chunking in a way that gcp, azure can also use can be deffered to another point in time when we are implementing those constructs. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;

Review Comment:
   This is an optional config. Moreover, the range is controlled by AWS s3. https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
   
   As the consumers of this config, work on top of streams, we do not know the full size of the object hence this is required. 
   
   Also if the user does not specify it, we calculate it from the maxResultSize.
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3StorageConnectorModule.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.inject.Binder;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.Collections;
+import java.util.List;
+
+public class S3StorageConnectorModule implements DruidModule
+{
+  @Override
+  public List<? extends Module> getJacksonModules()
+  {
+    return Collections.singletonList(
+        new SimpleModule(this.getClass().getSimpleName()).registerSubtypes(S3StorageConnectorProvider.class)
+    );
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+  }

Review Comment:
   So connector would be a namespace based singleton in guice which will have items like configs injected to it. 
   Lifecycle actually only makes sense for local storage which can be tackled as part of a follow up PR. 
   Each connector can have a custom config that has more things injected into it. 
   All config based user inputs like bucket, prefix are injected via Json configuration and rest like s3 client are injected using guice. 
   
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig

Review Comment:
   Nope, each S3OutputConfig is namespaced by the extension. So we might have multiple output configs per druid process but only one would be activated per extension. 



##########
processing/src/main/java/org/apache/druid/guice/StartupInjectorBuilder.java:
##########
@@ -47,18 +48,15 @@ public StartupInjectorBuilder()
         new JacksonModule(),
         new ConfigModule(),
         new NullHandlingModule(),
+        new StorageConnectorModule(),

Review Comment:
   Yeah, the CoreInjector seems the correct place.
   I have added the comment in the StartupInjectorBuilder class. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;

Review Comment:
   These come from the user as part of the extension config. 
   Checkout the comment in storageConnector interface. 



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;

Review Comment:
   Added the underscore representation. 



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;
+    chunkSize = 9000000L;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[9000000] is too small for maxResultsSize[100000000000]. chunkSize should be at least [10000000]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooSmallChunkSizeMaxResultsSizeIsNotRetionalToMaxPartNum() throws IOException
+  {
+    maxResultsSize = 274877906944L;
+    chunkSize = 27487790;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[27487790] is too small for maxResultsSize[274877906944]. chunkSize should be at least [27487791]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooLargeChunkSize() throws IOException
+  {
+    maxResultsSize = 1024L * 1024 * 1024 * 1024;
+    chunkSize = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE + 1;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[5368709121] should be smaller than [5368709120]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testWriteAndHappy() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testWriteSizeLargerThanConfiguredMaxChunkSizeShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES * 3);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      bb.clear();
+      bb.putInt(1);
+      bb.putInt(2);
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+    // each chunk 10 bytes, so there should be 2 chunks.
+    Assert.assertEquals(2, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 3);
+  }
+
+  @Test
+  public void testWriteSmallBufferShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 128;
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 600; i++) {
+        out.write(i);
+      }
+    }
+    // each chunk 128 bytes, so there should be 5 chunks.
+    Assert.assertEquals(5, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, 600);
+  }
+
+  @Test
+  public void testHitResultsSizeLimit() throws IOException
+  {
+    maxResultsSize = 50;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 14; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      Assert.assertThrows(
+          "Exceeded max results size [50]",
+          IOException.class,
+          () -> {
+            bb.clear();
+            bb.putInt(14);
+            out.write(bb.array());
+          }
+      );
+    }
+
+    s3.assertAborted();
+  }
+
+  @Test
+  public void testSuccessToUploadAfterRetry() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(1);
+
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testFailToUploadAfterRetries() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(3);
+
+    maxResultsSize = 1000;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 2; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      expectedException.expect(RuntimeException.class);
+      expectedException.expectCause(CoreMatchers.instanceOf(AmazonClientException.class));
+      expectedException.expectMessage("Upload failure test. Remaining failures [1]");
+      bb.clear();
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+
+    s3.assertAborted();
+  }
+
+  private static class TestAmazonS3 extends ServerSideEncryptingAmazonS3
+  {
+    private final List<UploadPartRequest> partRequests = new ArrayList<>();
+
+    private int uploadFailuresLeft;
+    private boolean aborted = false;
+    @Nullable
+    private CompleteMultipartUploadRequest completeRequest;
+
+    private TestAmazonS3(int totalUploadFailures)
+    {
+      super(EasyMock.createMock(AmazonS3.class), new NoopServerSideEncryption());

Review Comment:
   The purpose of this class is to act as a hook so that no calls are getting through to s3.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943805075


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")

Review Comment:
   I believe that IntelliJ inspections will complain if an implementation includes a `throws` unnecessarily, but I'm not sure about the interface: it can't know that third-part code, outside the code it inspects, might implement the `throws`.



##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  @Rule
+  TemporaryFolder temporaryFolder = new TemporaryFolder();
+  private File tempDir = temporaryFolder.newFolder();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir).get();
+
+  public LocalFileStorageConnectorTest() throws IOException {}

Review Comment:
   This is not needed, and will fail both checkstyle and IntelliJ inspections, sadly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943810311


##########
core/src/test/java/org/apache/druid/storage/local/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import org.apache.druid.storage.StorageConnector;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  @Rule
+  TemporaryFolder temporaryFolder = new TemporaryFolder();
+  private File tempDir = temporaryFolder.newFolder();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir).get();
+
+  public LocalFileStorageConnectorTest() throws IOException {}

Review Comment:
   as tempFolder.newFolder() throws an exception, the constructor is maked with this but I do see that our checkstyle would most likely fail. 
   Adjusted the code to make it pass.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on a diff in pull request #12874: Introducing Storage connector Inteface

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r940821657


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.

Review Comment:
   Maybe an explanation about the inherent race condition on a distributed system? To avoid that, presumably the writer should create a subdirectory just for this task?



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector

Review Comment:
   Nit: a bit of Javadoc to explain this? Like the fact that it must be configured with a base path and, presumably, all other paths are relative to this path? Is that true even if the read/write path is something like "/tmp/foo" that is, itself, an absolute path.



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch

Review Comment:
   I know we're not big into Javadoc formatting, but the above will appear jumbled on a single line when viewed in an IDE using the Javadoc viewer. Maybe at least put <pre></pre> around the code block?



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector
+{
+
+  public final String basePath;

Review Comment:
   Is `public` intentional?



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector
+{
+
+  public final String basePath;
+
+  public LocalFileStorageConnector(String basePath) throws IOException
+  {
+    this.basePath = basePath;
+    FileUtils.mkdirp(new File(basePath));
+  }
+
+  @Override
+  public boolean pathExists(String path)
+  {
+    return new File(objectPath(path)).exists();
+  }
+
+  @Override
+  public InputStream read(String path) throws IOException
+  {
+    return Files.newInputStream(new File(objectPath(path)).toPath());
+  }
+
+  @Override
+  public OutputStream write(String path) throws IOException
+  {
+    File toWrite = new File(objectPath(path));
+    FileUtils.mkdirp(toWrite.getParentFile());
+    return Files.newOutputStream(toWrite.toPath());
+  }
+
+  @Override
+  public void delete(String path) throws IOException
+  {
+    Files.delete(new File(objectPath(path)).toPath());
+  }
+
+  @Override
+  public void deleteRecursively(String dirName) throws IOException
+  {
+    FileUtils.deleteDirectory(new File(objectPath(dirName)));
+  }
+
+  private String objectPath(String path)

Review Comment:
   Here we're using routines that take a `File` object. Should we return that rather than going back & forth from strings to files? That is:
   
   ```java
   private File objectPath(String path)
   ```



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnector.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class LocalFileStorageConnector implements StorageConnector
+{
+
+  public final String basePath;
+
+  public LocalFileStorageConnector(String basePath) throws IOException
+  {
+    this.basePath = basePath;
+    FileUtils.mkdirp(new File(basePath));

Review Comment:
   Suppose this is for temporary files. Do we need the ability to remove the directory when we're done? Or, should we require the that directory exists already?



##########
core/src/main/java/org/apache/druid/storage/StorageConnectorModule.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Binder;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.List;
+
+public class StorageConnectorModule implements DruidModule
+{
+  @Override
+  public List<? extends Module> getJacksonModules()
+  {
+    return ImmutableList.of(new SimpleModule(StorageConnector.class.getSimpleName()).registerSubtypes(
+        LocalFileStorageConnectorProvider.class));
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+

Review Comment:
   Should we provide an out-of-the-box binding for things like a temporary local folder that can be configured via properties? Maybe not in this PR, since we'd have to change stuff that already handles temp files?



##########
core/src/main/java/org/apache/druid/storage/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.ISE;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new ISE(
+          e,
+          "Unable to create storageConnector[%s] for basepath[%s]",

Review Comment:
   Nit: space before `[`: "Unable to create storageConnector [Foo] for base path [/tmp]".
   
   Also, since this is for humans, "storage connector" as two words.



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();
+    String uuid2 = uuid_base + "/" + UUID.randomUUID();
+
+    createFile(uuid1);
+    createFile(uuid2);
+
+    Assert.assertTrue(storageConnector.pathExists(uuid1));
+    Assert.assertTrue(storageConnector.pathExists(uuid2));
+
+    checkContents(uuid1);
+    checkContents(uuid2);
+
+    File baseFile = new File(tempDir.getAbsolutePath() + "/" + uuid_base);

Review Comment:
   ```java
   File baseFile = new File(new File(tempDir.getAbsolutePath()), uuid_base);
   ```
   
   Here and the other places we do our own directory path building.



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();

Review Comment:
   Prefer the [JUnit `TemporaryDirectory` solution](https://junit.org/junit4/javadoc/latest/org/junit/rules/TemporaryFolder.html) rather than roll-your-own:
   
   ```java
     @ClassRule
     public static TemporaryFolder tempDir = new TemporaryFolder();
     ...
     File testDir = tempDir.newFolder();
   ```
   
   Oddly, a later test *does*, in fact, use the JUnit mechanism.



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   As it turns out, the HDFS client has this same concept. The [HDFS form](https://hdfscli.readthedocs.io/en/latest/api.html) is overkill for our needs, but it does reflect what distributed systems turn out to need. Another reference is the [`DrillFileSystem` class in Drill.](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java).
   
   Though, I do think you're on the right path by having this "low-level" interface separate from higher-level tasks. The examples above mix the two which ends up with much redundancy. The segment pusher is an example of such a higher-level utility.



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();
+    String uuid2 = uuid_base + "/" + UUID.randomUUID();
+
+    createFile(uuid1);
+    createFile(uuid2);
+
+    Assert.assertTrue(storageConnector.pathExists(uuid1));
+    Assert.assertTrue(storageConnector.pathExists(uuid2));
+
+    checkContents(uuid1);
+    checkContents(uuid2);
+
+    File baseFile = new File(tempDir.getAbsolutePath() + "/" + uuid_base);
+    Assert.assertTrue(baseFile.exists());
+    Assert.assertTrue(baseFile.isDirectory());
+    Assert.assertEquals(2, baseFile.listFiles().length);
+
+    storageConnector.deleteRecursively(uuid_base);
+    Assert.assertFalse(baseFile.exists());
+
+  }
+
+  private void checkContents(String uuid) throws IOException
+  {
+    InputStream inputStream = storageConnector.read(uuid);
+    Assert.assertEquals(1, inputStream.read());
+    Assert.assertEquals(0, inputStream.available());
+    inputStream.close();
+  }
+
+  private void createFile(String uuid) throws IOException
+  {
+    OutputStream os = storageConnector.write(uuid);
+    os.write(1);
+    os.close();

Review Comment:
   The "modern" approach is try-with-resources:
   
   ```java
   try (OutputStream os = storageConnector.write(uuid)) {
     os.write(1)
   }
   ```
   
   And above on read.



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());

Review Comment:
   See below.



##########
core/src/test/java/org/apache/druid/storage/LocalFileStorageConnectorTest.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.java.util.common.FileUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.UUID;
+
+public class LocalFileStorageConnectorTest
+{
+  private File tempDir = FileUtils.createTempDir();
+  private StorageConnector storageConnector = new LocalFileStorageConnectorProvider(tempDir.getPath()).get();
+
+  public LocalFileStorageConnectorTest()
+  {
+    tempDir.deleteOnExit();
+  }
+
+  @Test
+  public void sanityCheck() throws IOException
+  {
+    String uuid = UUID.randomUUID().toString();
+
+    //create file
+    createFile(uuid);
+
+    // check if file is created
+    Assert.assertTrue(storageConnector.pathExists(uuid));
+    Assert.assertTrue(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+
+    // check contents
+    checkContents(uuid);
+
+    // delete file
+    storageConnector.delete(uuid);
+    Assert.assertFalse(new File(tempDir.getAbsolutePath() + "/" + uuid).exists());
+  }
+
+  @Test
+  public void deleteRecursivelyTest() throws IOException
+  {
+    String uuid_base = UUID.randomUUID().toString();
+    String uuid1 = uuid_base + "/" + UUID.randomUUID();
+    String uuid2 = uuid_base + "/" + UUID.randomUUID();

Review Comment:
   By using the JUnit temp folder, you can use more meaningful names since JUnit will ensure that the temp directory is unique.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;

Review Comment:
   Please include units: `...SIZE_BYTES`.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;

Review Comment:
   This seems misplaced. We likely need a temp dir. But, we should compute it from some general temp dir and a subdir we compute. Else, we've got multiple temp dirs to sort out.
   
   Also, each output needs its own temp dir, so names don't collide, else the writer has to ensure unique file names, which is a nuisance. Better is for each use that needs a temp dir to create a temporary subdir, dnd ensure it is cleaned up afterwards.
   
   Extra bonus: suppose the power goes out. How do we clean up in-flight temp files on Druid restart?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId
+    );
+  }
+
+  @Override
+  public void write(int b) throws IOException
+  {
+    singularBuffer[0] = (byte) b;
+    write(singularBuffer, 0, 1);
+  }
+
+  @Override
+  public void write(byte[] b, int off, int len) throws IOException
+  {
+    if (b == null) {
+      error = true;
+      throw new NullPointerException();
+    } else if ((off < 0) || (off > b.length) || (len < 0) ||
+               ((off + len) > b.length) || ((off + len) < 0)) {
+      error = true;
+      throw new IndexOutOfBoundsException();
+    } else if (len == 0) {
+      return;
+    }
+
+    try {
+      int offsetToWrite = off;
+      int remainingBytesToWrite = len;
+
+      while (remainingBytesToWrite > 0) {
+        final int writtenBytes = writeToCurrentChunk(b, offsetToWrite, remainingBytesToWrite);
+
+        if (currentChunk.length() >= chunkSize) {
+          pushCurrentChunk();
+          currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+        }
+
+        offsetToWrite += writtenBytes;
+        remainingBytesToWrite -= writtenBytes;
+      }
+    }
+    catch (RuntimeException | IOException e) {
+      error = true;
+      throw e;
+    }
+  }
+
+  private int writeToCurrentChunk(byte[] b, int off, int len) throws IOException
+  {
+    final int lenToWrite = Math.min(len, Math.toIntExact(chunkSize - currentChunk.length()));
+    currentChunk.outputStream.write(b, off, lenToWrite);
+    return lenToWrite;
+  }
+
+  private void pushCurrentChunk() throws IOException
+  {
+    currentChunk.close();
+    final Chunk chunk = currentChunk;
+    try {
+      if (chunk.length() > 0) {
+        resultsSize += chunk.length();
+        if (resultsSize > config.getMaxResultsSize()) {
+          throw new IOE("Exceeded max results size [%s]", config.getMaxResultsSize());
+        }
+
+        pushStopwatch.start();
+        pushResults.add(push(chunk));
+        pushStopwatch.stop();
+        numChunksPushed++;
+      }
+    }
+    finally {
+      if (!chunk.delete()) {
+        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
+      }
+    }
+  }
+
+  private PartETag push(Chunk chunk) throws IOException
+  {
+    try {
+      return RetryUtils.retry(
+          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
+          S3Utils.S3RETRY,
+          config.getMaxTriesOnTransientError()
+      );
+    }
+    catch (AmazonServiceException e) {
+      throw new IOException(e);
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private PartETag uploadPartIfPossible(
+      String uploadId,
+      String bucket,
+      String key,
+      Chunk chunk
+  )
+  {
+    final ObjectMetadata objectMetadata = new ObjectMetadata();
+    objectMetadata.setContentLength(resultsSize);
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunk.file)
+        .withPartNumber(chunk.id)
+        .withPartSize(chunk.length());
+
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunk, bucket, key);
+    }
+    UploadPartResult uploadResult = s3.uploadPart(uploadPartRequest);
+    return uploadResult.getPartETag();
+  }
+
+  @Override
+  public void close() throws IOException
+  {
+    if (closed) {
+      return;
+    }
+    closed = true;
+    Closer closer = Closer.create();
+
+    // Closeables are closed in LIFO order
+    closer.register(() -> {
+      // This should be emitted as a metric
+      LOG.info("Total push time: [%s] ms", pushStopwatch.elapsed(TimeUnit.MILLISECONDS));

Review Comment:
   `%s` for a number? And, since this is a number, we don't really need the Druid-style quotes (i.e. brackets)



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");

Review Comment:
   Are we mixing metaphors here? This class is called `S3OutputConfig` which seems pretty generic. But, here we're talking about query results, which seems pretty specific. (I'd have expected an "S3 Output" to talk about "objects", which is what S3 manages.)
   
   If this is *only* for queries, then shouldn't this be called `S3QueryResultConfig` or some such?
   
   And, should we document in code that all queries, for all users, across all time, must go into the same bucket?
   
   Also, since we don't really have query results for `SELECT` queries, can we discuss what "query results" are? Are these intermediate data sets within a task? If so, they are not really "results" so much as a `S3StageResultsCache`, if that's what this is.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId
+    );
+  }
+
+  @Override
+  public void write(int b) throws IOException
+  {
+    singularBuffer[0] = (byte) b;
+    write(singularBuffer, 0, 1);
+  }
+
+  @Override
+  public void write(byte[] b, int off, int len) throws IOException
+  {
+    if (b == null) {
+      error = true;
+      throw new NullPointerException();
+    } else if ((off < 0) || (off > b.length) || (len < 0) ||
+               ((off + len) > b.length) || ((off + len) < 0)) {
+      error = true;
+      throw new IndexOutOfBoundsException();
+    } else if (len == 0) {
+      return;
+    }
+
+    try {
+      int offsetToWrite = off;
+      int remainingBytesToWrite = len;
+
+      while (remainingBytesToWrite > 0) {
+        final int writtenBytes = writeToCurrentChunk(b, offsetToWrite, remainingBytesToWrite);
+
+        if (currentChunk.length() >= chunkSize) {
+          pushCurrentChunk();
+          currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+        }
+
+        offsetToWrite += writtenBytes;
+        remainingBytesToWrite -= writtenBytes;
+      }
+    }
+    catch (RuntimeException | IOException e) {
+      error = true;
+      throw e;
+    }
+  }
+
+  private int writeToCurrentChunk(byte[] b, int off, int len) throws IOException
+  {
+    final int lenToWrite = Math.min(len, Math.toIntExact(chunkSize - currentChunk.length()));
+    currentChunk.outputStream.write(b, off, lenToWrite);
+    return lenToWrite;
+  }
+
+  private void pushCurrentChunk() throws IOException
+  {
+    currentChunk.close();
+    final Chunk chunk = currentChunk;
+    try {
+      if (chunk.length() > 0) {
+        resultsSize += chunk.length();
+        if (resultsSize > config.getMaxResultsSize()) {
+          throw new IOE("Exceeded max results size [%s]", config.getMaxResultsSize());
+        }
+
+        pushStopwatch.start();
+        pushResults.add(push(chunk));
+        pushStopwatch.stop();
+        numChunksPushed++;
+      }
+    }
+    finally {
+      if (!chunk.delete()) {
+        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
+      }
+    }
+  }
+
+  private PartETag push(Chunk chunk) throws IOException
+  {
+    try {
+      return RetryUtils.retry(
+          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
+          S3Utils.S3RETRY,
+          config.getMaxTriesOnTransientError()
+      );
+    }
+    catch (AmazonServiceException e) {
+      throw new IOException(e);
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private PartETag uploadPartIfPossible(
+      String uploadId,
+      String bucket,
+      String key,
+      Chunk chunk
+  )
+  {
+    final ObjectMetadata objectMetadata = new ObjectMetadata();
+    objectMetadata.setContentLength(resultsSize);
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunk.file)
+        .withPartNumber(chunk.id)
+        .withPartSize(chunk.length());
+
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunk, bucket, key);
+    }
+    UploadPartResult uploadResult = s3.uploadPart(uploadPartRequest);
+    return uploadResult.getPartETag();
+  }
+
+  @Override
+  public void close() throws IOException
+  {
+    if (closed) {
+      return;
+    }
+    closed = true;
+    Closer closer = Closer.create();
+
+    // Closeables are closed in LIFO order
+    closer.register(() -> {
+      // This should be emitted as a metric
+      LOG.info("Total push time: [%s] ms", pushStopwatch.elapsed(TimeUnit.MILLISECONDS));
+    });
+
+    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+
+    closer.register(() -> {
+      try {
+        if (resultsSize > 0 && isAllPushSucceeded()) {

Review Comment:
   This logic might be better represented as a `flush()` method which is called here.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId
+    );
+  }
+
+  @Override
+  public void write(int b) throws IOException
+  {
+    singularBuffer[0] = (byte) b;
+    write(singularBuffer, 0, 1);
+  }
+
+  @Override
+  public void write(byte[] b, int off, int len) throws IOException
+  {
+    if (b == null) {
+      error = true;
+      throw new NullPointerException();
+    } else if ((off < 0) || (off > b.length) || (len < 0) ||
+               ((off + len) > b.length) || ((off + len) < 0)) {
+      error = true;
+      throw new IndexOutOfBoundsException();
+    } else if (len == 0) {
+      return;
+    }
+
+    try {
+      int offsetToWrite = off;
+      int remainingBytesToWrite = len;
+
+      while (remainingBytesToWrite > 0) {
+        final int writtenBytes = writeToCurrentChunk(b, offsetToWrite, remainingBytesToWrite);
+
+        if (currentChunk.length() >= chunkSize) {
+          pushCurrentChunk();
+          currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+        }
+
+        offsetToWrite += writtenBytes;
+        remainingBytesToWrite -= writtenBytes;
+      }
+    }
+    catch (RuntimeException | IOException e) {
+      error = true;
+      throw e;
+    }
+  }
+
+  private int writeToCurrentChunk(byte[] b, int off, int len) throws IOException
+  {
+    final int lenToWrite = Math.min(len, Math.toIntExact(chunkSize - currentChunk.length()));
+    currentChunk.outputStream.write(b, off, lenToWrite);
+    return lenToWrite;
+  }
+
+  private void pushCurrentChunk() throws IOException
+  {
+    currentChunk.close();
+    final Chunk chunk = currentChunk;
+    try {
+      if (chunk.length() > 0) {
+        resultsSize += chunk.length();
+        if (resultsSize > config.getMaxResultsSize()) {
+          throw new IOE("Exceeded max results size [%s]", config.getMaxResultsSize());
+        }
+
+        pushStopwatch.start();
+        pushResults.add(push(chunk));
+        pushStopwatch.stop();
+        numChunksPushed++;
+      }
+    }
+    finally {
+      if (!chunk.delete()) {
+        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
+      }
+    }
+  }
+
+  private PartETag push(Chunk chunk) throws IOException
+  {
+    try {
+      return RetryUtils.retry(
+          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
+          S3Utils.S3RETRY,
+          config.getMaxTriesOnTransientError()
+      );
+    }
+    catch (AmazonServiceException e) {
+      throw new IOException(e);
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private PartETag uploadPartIfPossible(
+      String uploadId,
+      String bucket,
+      String key,
+      Chunk chunk
+  )
+  {
+    final ObjectMetadata objectMetadata = new ObjectMetadata();
+    objectMetadata.setContentLength(resultsSize);
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunk.file)
+        .withPartNumber(chunk.id)
+        .withPartSize(chunk.length());
+
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunk, bucket, key);
+    }
+    UploadPartResult uploadResult = s3.uploadPart(uploadPartRequest);
+    return uploadResult.getPartETag();
+  }
+
+  @Override
+  public void close() throws IOException
+  {
+    if (closed) {
+      return;
+    }
+    closed = true;
+    Closer closer = Closer.create();
+
+    // Closeables are closed in LIFO order
+    closer.register(() -> {
+      // This should be emitted as a metric
+      LOG.info("Total push time: [%s] ms", pushStopwatch.elapsed(TimeUnit.MILLISECONDS));
+    });
+
+    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+
+    closer.register(() -> {
+      try {
+        if (resultsSize > 0 && isAllPushSucceeded()) {
+          RetryUtils.retry(
+              () -> s3.completeMultipartUpload(
+                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
+              ),
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        } else {
+          RetryUtils.retry(
+              () -> {
+                s3.abortMultipartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
+                return null;
+              },
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        }
+      }
+      catch (Exception e) {
+        throw new IOException(e);
+      }
+    });
+
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+      }
+    }
+  }
+
+  private boolean isAllPushSucceeded()
+  {
+    return !error && !pushResults.isEmpty() && numChunksPushed == pushResults.size();
+  }
+
+  private static class Chunk implements Closeable
+  {
+    private final int id;
+    private final File file;
+    private final CountingOutputStream outputStream;
+    private boolean closed;
+
+    private Chunk(int id, File file) throws FileNotFoundException
+    {
+      this.id = id;
+      this.file = file;
+      this.outputStream = new CountingOutputStream(new FastBufferedOutputStream(new FileOutputStream(file)));
+    }
+
+    private long length()
+    {
+      return outputStream.getCount();
+    }
+
+    private boolean delete()
+    {
+      return file.delete();
+    }
+
+    private String getAbsolutePath()
+    {
+      return file.getAbsolutePath();
+    }
+
+    @Override
+    public boolean equals(Object o)

Review Comment:
   Why would chunks be compared for equality? Under what case would we create two chunks and expect the to be the same?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;

Review Comment:
   This is a config. Shouldn't we specify a single chunk size? Else, what does it mean to have a range? How big should the range be? Do we try to fit chunks into this size, or does the caller have to create these chunks? Best is if the user just provides data, and this class creates the size of chunks specified here.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId
+    );
+  }
+
+  @Override
+  public void write(int b) throws IOException
+  {
+    singularBuffer[0] = (byte) b;
+    write(singularBuffer, 0, 1);
+  }
+
+  @Override
+  public void write(byte[] b, int off, int len) throws IOException
+  {
+    if (b == null) {
+      error = true;
+      throw new NullPointerException();
+    } else if ((off < 0) || (off > b.length) || (len < 0) ||
+               ((off + len) > b.length) || ((off + len) < 0)) {
+      error = true;
+      throw new IndexOutOfBoundsException();
+    } else if (len == 0) {
+      return;
+    }
+
+    try {
+      int offsetToWrite = off;
+      int remainingBytesToWrite = len;
+
+      while (remainingBytesToWrite > 0) {
+        final int writtenBytes = writeToCurrentChunk(b, offsetToWrite, remainingBytesToWrite);
+
+        if (currentChunk.length() >= chunkSize) {
+          pushCurrentChunk();
+          currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+        }
+
+        offsetToWrite += writtenBytes;
+        remainingBytesToWrite -= writtenBytes;
+      }
+    }
+    catch (RuntimeException | IOException e) {
+      error = true;
+      throw e;
+    }
+  }
+
+  private int writeToCurrentChunk(byte[] b, int off, int len) throws IOException
+  {
+    final int lenToWrite = Math.min(len, Math.toIntExact(chunkSize - currentChunk.length()));
+    currentChunk.outputStream.write(b, off, lenToWrite);
+    return lenToWrite;
+  }
+
+  private void pushCurrentChunk() throws IOException
+  {
+    currentChunk.close();
+    final Chunk chunk = currentChunk;
+    try {
+      if (chunk.length() > 0) {
+        resultsSize += chunk.length();
+        if (resultsSize > config.getMaxResultsSize()) {
+          throw new IOE("Exceeded max results size [%s]", config.getMaxResultsSize());
+        }
+
+        pushStopwatch.start();
+        pushResults.add(push(chunk));
+        pushStopwatch.stop();
+        numChunksPushed++;
+      }
+    }
+    finally {
+      if (!chunk.delete()) {
+        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
+      }
+    }
+  }
+
+  private PartETag push(Chunk chunk) throws IOException
+  {
+    try {
+      return RetryUtils.retry(
+          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
+          S3Utils.S3RETRY,
+          config.getMaxTriesOnTransientError()
+      );
+    }
+    catch (AmazonServiceException e) {
+      throw new IOException(e);
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private PartETag uploadPartIfPossible(
+      String uploadId,
+      String bucket,
+      String key,
+      Chunk chunk
+  )
+  {
+    final ObjectMetadata objectMetadata = new ObjectMetadata();
+    objectMetadata.setContentLength(resultsSize);
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunk.file)
+        .withPartNumber(chunk.id)
+        .withPartSize(chunk.length());
+
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunk, bucket, key);
+    }
+    UploadPartResult uploadResult = s3.uploadPart(uploadPartRequest);
+    return uploadResult.getPartETag();
+  }
+
+  @Override
+  public void close() throws IOException
+  {
+    if (closed) {
+      return;
+    }
+    closed = true;
+    Closer closer = Closer.create();
+
+    // Closeables are closed in LIFO order
+    closer.register(() -> {
+      // This should be emitted as a metric
+      LOG.info("Total push time: [%s] ms", pushStopwatch.elapsed(TimeUnit.MILLISECONDS));
+    });
+
+    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+
+    closer.register(() -> {
+      try {
+        if (resultsSize > 0 && isAllPushSucceeded()) {
+          RetryUtils.retry(
+              () -> s3.completeMultipartUpload(
+                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
+              ),
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        } else {
+          RetryUtils.retry(
+              () -> {
+                s3.abortMultipartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
+                return null;
+              },
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        }
+      }
+      catch (Exception e) {
+        throw new IOException(e);
+      }
+    });
+
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+      }
+    }
+  }
+
+  private boolean isAllPushSucceeded()
+  {
+    return !error && !pushResults.isEmpty() && numChunksPushed == pushResults.size();
+  }
+
+  private static class Chunk implements Closeable
+  {
+    private final int id;
+    private final File file;
+    private final CountingOutputStream outputStream;
+    private boolean closed;
+
+    private Chunk(int id, File file) throws FileNotFoundException
+    {
+      this.id = id;
+      this.file = file;
+      this.outputStream = new CountingOutputStream(new FastBufferedOutputStream(new FileOutputStream(file)));
+    }
+
+    private long length()
+    {
+      return outputStream.getCount();
+    }
+
+    private boolean delete()
+    {
+      return file.delete();

Review Comment:
   Should we allow deletion while the stream is open? Otherwise, what does it mean to delete the file midway? Actually, why does a stream have a sense of a file? Shouldn't those be separate concepts?
   
   Since this class is private, would it make more sense to delete the file on close?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;

Review Comment:
   Where do the credentials come from? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");
+
+  /**
+   * Max number of tries for each upload.
+   */
+  @JsonProperty
+  private int maxTriesOnTransientErrors = RetryUtils.DEFAULT_MAX_TRIES;
+
+  public String getBucket()
+  {
+    return bucket;
+  }
+
+  public String getPrefix()
+  {
+    return prefix;
+  }
+
+  public File getTempDir()
+  {
+    return tempDir;
+  }
+
+  @Nullable
+  public Long getChunkSize()
+  {
+    return chunkSize == null ? null : chunkSize.getBytes();

Review Comment:
   Should this class provide a default? Else, every caller has to know of some default if this value is null.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3StorageConnectorModule.java:
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.inject.Binder;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.Collections;
+import java.util.List;
+
+public class S3StorageConnectorModule implements DruidModule
+{
+  @Override
+  public List<? extends Module> getJacksonModules()
+  {
+    return Collections.singletonList(
+        new SimpleModule(this.getClass().getSimpleName()).registerSubtypes(S3StorageConnectorProvider.class)
+    );
+  }
+
+  @Override
+  public void configure(Binder binder)
+  {
+  }

Review Comment:
   Trying to sort out how this works. I guess I'd expect the "connector" to be a singleton in Guice. That connector can have items injected into it, including its config. It might have a lifecycle, so it can delete files on start and exit. It might have a secrets provider, such as for S3. Then each request can open a file, say, based on that connector, using maybe a "connection" or a "session". That session would also delete its files on session close. The session can give rise to streams or chunked operations.
   
   Something like that is the usual way these things work. I'm wondering if the JSON configuration gives us the flexibility these kinds of things usually need.
   
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig

Review Comment:
   Since this is a config, and configs are created from JSON properties, does this mean that there is only one S3 output config per Druid process? Is that the right semantics?
   
   If so, maybe a comment to say that this is the output for... what? That all tasks/uploads/pushes/whatevers have to go into the same bucket?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId

Review Comment:
   Ah! So this is only for Async?
   
   Since we're at a lower level, should this be generalized? For anything that wants to write to S3? And automate the math?



##########
processing/src/main/java/org/apache/druid/guice/StartupInjectorBuilder.java:
##########
@@ -47,18 +48,15 @@ public StartupInjectorBuilder()
         new JacksonModule(),
         new ConfigModule(),
         new NullHandlingModule(),
+        new StorageConnectorModule(),
         new ExpressionProcessingModule(),
-        binder -> {
-          binder.bind(DruidSecondaryModule.class);
-        }
+        binder -> binder.bind(DruidSecondaryModule.class)

Review Comment:
   Thanks for the cleanup. I guess I expected to find more stuff that would go here, but never did...



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",
+          chunkSize,
+          maxResultsSize,
+          computeMinChunkSize(maxResultsSize)
+      );
+    }
+    if (S3_MULTIPART_UPLOAD_MAX_PART_SIZE < chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] should be smaller than [%s]",
+          chunkSize,
+          S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+      );
+    }
+  }
+
+  private static long computeMinChunkSize(long maxResultsSize)
+  {
+    return Math.max(
+        (long) Math.ceil(maxResultsSize / (double) S3_MULTIPART_UPLOAD_MAX_NUM_PARTS),
+        S3_MULTIPART_UPLOAD_MIN_PART_SIZE
+    );
+  }
+
+  public static String getS3KeyForQuery(String prefix, String asyncResultId)
+  {
+    return JOINER.join(
+        prefix,
+        asyncResultId
+    );
+  }
+
+  @Override
+  public void write(int b) throws IOException
+  {
+    singularBuffer[0] = (byte) b;
+    write(singularBuffer, 0, 1);
+  }
+
+  @Override
+  public void write(byte[] b, int off, int len) throws IOException
+  {
+    if (b == null) {
+      error = true;
+      throw new NullPointerException();
+    } else if ((off < 0) || (off > b.length) || (len < 0) ||
+               ((off + len) > b.length) || ((off + len) < 0)) {
+      error = true;
+      throw new IndexOutOfBoundsException();
+    } else if (len == 0) {
+      return;
+    }
+
+    try {
+      int offsetToWrite = off;
+      int remainingBytesToWrite = len;
+
+      while (remainingBytesToWrite > 0) {
+        final int writtenBytes = writeToCurrentChunk(b, offsetToWrite, remainingBytesToWrite);
+
+        if (currentChunk.length() >= chunkSize) {
+          pushCurrentChunk();
+          currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+        }
+
+        offsetToWrite += writtenBytes;
+        remainingBytesToWrite -= writtenBytes;
+      }
+    }
+    catch (RuntimeException | IOException e) {
+      error = true;
+      throw e;
+    }
+  }
+
+  private int writeToCurrentChunk(byte[] b, int off, int len) throws IOException
+  {
+    final int lenToWrite = Math.min(len, Math.toIntExact(chunkSize - currentChunk.length()));
+    currentChunk.outputStream.write(b, off, lenToWrite);
+    return lenToWrite;
+  }
+
+  private void pushCurrentChunk() throws IOException
+  {
+    currentChunk.close();
+    final Chunk chunk = currentChunk;
+    try {
+      if (chunk.length() > 0) {
+        resultsSize += chunk.length();
+        if (resultsSize > config.getMaxResultsSize()) {
+          throw new IOE("Exceeded max results size [%s]", config.getMaxResultsSize());
+        }
+
+        pushStopwatch.start();
+        pushResults.add(push(chunk));
+        pushStopwatch.stop();
+        numChunksPushed++;
+      }
+    }
+    finally {
+      if (!chunk.delete()) {
+        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
+      }
+    }
+  }
+
+  private PartETag push(Chunk chunk) throws IOException
+  {
+    try {
+      return RetryUtils.retry(
+          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
+          S3Utils.S3RETRY,
+          config.getMaxTriesOnTransientError()
+      );
+    }
+    catch (AmazonServiceException e) {
+      throw new IOException(e);
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private PartETag uploadPartIfPossible(
+      String uploadId,
+      String bucket,
+      String key,
+      Chunk chunk
+  )
+  {
+    final ObjectMetadata objectMetadata = new ObjectMetadata();
+    objectMetadata.setContentLength(resultsSize);
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunk.file)
+        .withPartNumber(chunk.id)
+        .withPartSize(chunk.length());
+
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunk, bucket, key);
+    }
+    UploadPartResult uploadResult = s3.uploadPart(uploadPartRequest);
+    return uploadResult.getPartETag();
+  }
+
+  @Override
+  public void close() throws IOException
+  {
+    if (closed) {
+      return;
+    }
+    closed = true;
+    Closer closer = Closer.create();
+
+    // Closeables are closed in LIFO order
+    closer.register(() -> {
+      // This should be emitted as a metric
+      LOG.info("Total push time: [%s] ms", pushStopwatch.elapsed(TimeUnit.MILLISECONDS));
+    });
+
+    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+
+    closer.register(() -> {
+      try {
+        if (resultsSize > 0 && isAllPushSucceeded()) {
+          RetryUtils.retry(
+              () -> s3.completeMultipartUpload(
+                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
+              ),
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        } else {
+          RetryUtils.retry(
+              () -> {
+                s3.abortMultipartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
+                return null;
+              },
+              S3Utils.S3RETRY,
+              config.getMaxTriesOnTransientError()
+          );
+        }
+      }
+      catch (Exception e) {
+        throw new IOException(e);
+      }
+    });
+
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+      }
+    }

Review Comment:
   This seems somewhat odd. It would be clearer to push the current chunk first, and to explicitly call `closer.close()`. Or, if we want try-with-with-reasources (which seems overkill here), the try should include the closer creation.
   
   Is the closer actually buying us anything? Any reason to not just do the steps, in forward order, in this method? Or, is the goal here to catch & suppress errors?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.

Review Comment:
   See comment above about formatting.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.
+   */
+  private long resultsSize;
+
+  /**
+   * A flag indicating whether there was an upload error.
+   * This flag is tested in {@link #close()} to determine whether it needs to upload the current chunk or not.
+   */
+  private boolean error;
+  private boolean closed;
+
+  public RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key
+  ) throws IOException
+  {
+
+    this(config, s3, s3Key, true);
+  }
+
+  @VisibleForTesting
+  protected RetriableS3OutputStream(
+      S3OutputConfig config,
+      ServerSideEncryptingAmazonS3 s3,
+      String s3Key,
+      boolean chunkValidation
+  ) throws IOException
+  {
+    this.config = config;
+    this.s3 = s3;
+    this.s3Key = s3Key;
+
+    final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
+        new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+    );
+    this.uploadId = result.getUploadId();
+    this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
+    FileUtils.mkdirp(this.chunkStorePath);
+    this.chunkSize = config.getChunkSize() == null ? computeChunkSize(config) : config.getChunkSize();
+    if (chunkValidation) {
+      validateChunkSize(config.getMaxResultsSize(), chunkSize);
+    }
+    this.pushStopwatch = Stopwatch.createUnstarted();
+    this.pushStopwatch.reset();
+
+    this.currentChunk = new Chunk(nextChunkId, new File(chunkStorePath, String.valueOf(nextChunkId++)));
+  }
+
+  private static long computeChunkSize(S3OutputConfig config)
+  {
+    return computeMinChunkSize(config.getMaxResultsSize());
+  }
+
+  private static void validateChunkSize(long maxResultsSize, long chunkSize)
+  {
+    if (computeMinChunkSize(maxResultsSize) > chunkSize) {
+      throw new IAE(
+          "chunkSize[%s] is too small for maxResultsSize[%s]. chunkSize should be at least [%s]",

Review Comment:
   `%s` for a number. `%d` is preferred. Please add spaces before brackets, though brackets are not really needed for numbers.
   
   Maybe we can improve the message. "(What?) selected an S3 upload chunk size of %d bytes, but the configuration requires at least %d bytes, due to a configured maxResultSize of %d bytes."
   
   But, taking a step back, I'm not sure that I, as a user, would have any idea what to do. Why should I care about chunk size? How does the maximum file size relate to chunk size? More to the point, why didn't Druid do the math itself to pick the chunk size? Even more, if the chunks are too small, why can't Druid just concatenate several of them to get its preferred size?
   
   There is probably a reason. Can we maybe add some Javadoc to explain? Or, if this is copy/pasted from existing code, and that code is documented, perhaps point to the docs.



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   Often a tool wants an atomic replace, or the ability to create a temp file with a guaranteed unique name. Also the ability to list the contents of a directory. A very handy feature is a "delete on exit" form of open to track temporary files. The OS has built-in support, for things like S3 we have to emulate the behavior.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;
+    chunkSize = 9000000L;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[9000000] is too small for maxResultsSize[100000000000]. chunkSize should be at least [10000000]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooSmallChunkSizeMaxResultsSizeIsNotRetionalToMaxPartNum() throws IOException
+  {
+    maxResultsSize = 274877906944L;
+    chunkSize = 27487790;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[27487790] is too small for maxResultsSize[274877906944]. chunkSize should be at least [27487791]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooLargeChunkSize() throws IOException
+  {
+    maxResultsSize = 1024L * 1024 * 1024 * 1024;
+    chunkSize = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE + 1;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[5368709121] should be smaller than [5368709120]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testWriteAndHappy() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testWriteSizeLargerThanConfiguredMaxChunkSizeShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES * 3);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      bb.clear();
+      bb.putInt(1);
+      bb.putInt(2);
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+    // each chunk 10 bytes, so there should be 2 chunks.
+    Assert.assertEquals(2, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 3);
+  }
+
+  @Test
+  public void testWriteSmallBufferShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 128;
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 600; i++) {
+        out.write(i);
+      }
+    }
+    // each chunk 128 bytes, so there should be 5 chunks.
+    Assert.assertEquals(5, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, 600);
+  }
+
+  @Test
+  public void testHitResultsSizeLimit() throws IOException
+  {
+    maxResultsSize = 50;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(

Review Comment:
   Thanks for using try-with-resources.
   
   Having the client create the S3 output stream seems to couple the implementation to the client a bit too much. This is where that "connector" idea would be better:
   
   `OutputStream out = s3Connector.openForWrite(path)`. that way, the connector, not the client, manages the config and s3, and chooses the correct stream type. This lets us more easily improve the streams without locking the client into our implementation.



##########
processing/src/main/java/org/apache/druid/guice/StartupInjectorBuilder.java:
##########
@@ -47,18 +48,15 @@ public StartupInjectorBuilder()
         new JacksonModule(),
         new ConfigModule(),
         new NullHandlingModule(),
+        new StorageConnectorModule(),

Review Comment:
   Do we want this here? Does every test and client need this module? Does it need to be in the startup injector?
   
   Or, would this be better in the `forServer()` method of the `CoreInjectorBuilder`? If any connector needs to have anything injected, then it has to be part of the "second tier" of modules.
   
   The goal with these builders is to let test and clients use Guice, without being forced to also accept server-only components. So, for each new component, ask if tests (both UTs and ITs) need the component? Or, only the server?



##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the relative path exists in the underlying storage layer.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  boolean pathExists(String path) throws IOException;
+
+  /**
+   * Reads the data present at the relative path from the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return InputStream
+   * @throws IOException if the path is not present or the unable to read the data present on the path.
+   */
+  InputStream read(String path) throws IOException;
+
+  /**
+   * Open an {@link OutputStream} for writing data to the relative path in the underlying storage system.
+   * The caller should take care of closing the stream when done or in case of error.
+   *
+   * @param path
+   * @return
+   * @throws IOException
+   */
+  OutputStream write(String path) throws IOException;
+
+  /**
+   * Delete file present at the relative path.
+   *
+   * @param path
+   * @throws IOException
+   */
+  @SuppressWarnings("all")
+  void delete(String path) throws IOException;
+
+  /**
+   * Delete a directory pointed to by the path and also recursively deletes all files in said directory.
+   *
+   * @param path path
+   * @throws IOException
+   */
+  void deleteRecursively(String path) throws IOException;

Review Comment:
   Since we can delete a directory recursively, do we need a function to *create* that directory?



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;
+    chunkSize = 9000000L;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[9000000] is too small for maxResultsSize[100000000000]. chunkSize should be at least [10000000]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooSmallChunkSizeMaxResultsSizeIsNotRetionalToMaxPartNum() throws IOException
+  {
+    maxResultsSize = 274877906944L;
+    chunkSize = 27487790;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[27487790] is too small for maxResultsSize[274877906944]. chunkSize should be at least [27487791]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testTooLargeChunkSize() throws IOException
+  {
+    maxResultsSize = 1024L * 1024 * 1024 * 1024;
+    chunkSize = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE + 1;
+
+    expectedException.expect(IllegalArgumentException.class);
+    expectedException.expectMessage(
+        "chunkSize[5368709121] should be smaller than [5368709120]"
+    );
+    new RetriableS3OutputStream(config, s3, path).close();
+  }
+
+  @Test
+  public void testWriteAndHappy() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testWriteSizeLargerThanConfiguredMaxChunkSizeShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES * 3);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      bb.clear();
+      bb.putInt(1);
+      bb.putInt(2);
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+    // each chunk 10 bytes, so there should be 2 chunks.
+    Assert.assertEquals(2, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 3);
+  }
+
+  @Test
+  public void testWriteSmallBufferShouldSucceed() throws IOException
+  {
+    maxResultsSize = 1000;
+    chunkSize = 128;
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 600; i++) {
+        out.write(i);
+      }
+    }
+    // each chunk 128 bytes, so there should be 5 chunks.
+    Assert.assertEquals(5, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, 600);
+  }
+
+  @Test
+  public void testHitResultsSizeLimit() throws IOException
+  {
+    maxResultsSize = 50;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 14; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      Assert.assertThrows(
+          "Exceeded max results size [50]",
+          IOException.class,
+          () -> {
+            bb.clear();
+            bb.putInt(14);
+            out.write(bb.array());
+          }
+      );
+    }
+
+    s3.assertAborted();
+  }
+
+  @Test
+  public void testSuccessToUploadAfterRetry() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(1);
+
+    maxResultsSize = 1000;
+    chunkSize = 10;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 25; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+    }
+    // each chunk is 10 bytes, so there should be 10 chunks.
+    Assert.assertEquals(10, s3.partRequests.size());
+    s3.assertCompleted(chunkSize, Integer.BYTES * 25);
+  }
+
+  @Test
+  public void testFailToUploadAfterRetries() throws IOException
+  {
+    final TestAmazonS3 s3 = new TestAmazonS3(3);
+
+    maxResultsSize = 1000;
+    ByteBuffer bb = ByteBuffer.allocate(Integer.BYTES);
+    try (RetriableS3OutputStream out = new RetriableS3OutputStream(
+        config,
+        s3,
+        path,
+        false
+    )) {
+      for (int i = 0; i < 2; i++) {
+        bb.clear();
+        bb.putInt(i);
+        out.write(bb.array());
+      }
+
+      expectedException.expect(RuntimeException.class);
+      expectedException.expectCause(CoreMatchers.instanceOf(AmazonClientException.class));
+      expectedException.expectMessage("Upload failure test. Remaining failures [1]");
+      bb.clear();
+      bb.putInt(3);
+      out.write(bb.array());
+    }
+
+    s3.assertAborted();
+  }
+
+  private static class TestAmazonS3 extends ServerSideEncryptingAmazonS3
+  {
+    private final List<UploadPartRequest> partRequests = new ArrayList<>();
+
+    private int uploadFailuresLeft;
+    private boolean aborted = false;
+    @Nullable
+    private CompleteMultipartUploadRequest completeRequest;
+
+    private TestAmazonS3(int totalUploadFailures)
+    {
+      super(EasyMock.createMock(AmazonS3.class), new NoopServerSideEncryption());

Review Comment:
   If we mock S3, are we actually testing anything? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;
+
+  @Nullable
+  @JsonProperty
+  @HumanReadableBytesRange(
+      min = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MIN_PART_SIZE,
+      max = RetriableS3OutputStream.S3_MULTIPART_UPLOAD_MAX_PART_SIZE
+  ) // limits of s3 multipart upload
+  private HumanReadableBytes chunkSize;
+
+  /**
+   * Max size for each query results. This limit relies on the s3 multipart upload limits.
+   * See https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html for more details.
+   *
+   * @see RetriableS3OutputStream
+   */
+  @JsonProperty
+  @HumanReadableBytesRange(min = 5L * 1024 * 1024, max = 5L * 1024 * 1024 * 1024 * 1024)
+  private HumanReadableBytes maxResultsSize = new HumanReadableBytes("100MiB");
+
+  /**
+   * Max number of tries for each upload.
+   */
+  @JsonProperty
+  private int maxTriesOnTransientErrors = RetryUtils.DEFAULT_MAX_TRIES;
+
+  public String getBucket()
+  {
+    return bucket;
+  }
+
+  public String getPrefix()
+  {
+    return prefix;
+  }
+
+  public File getTempDir()
+  {
+    return tempDir;
+  }
+
+  @Nullable
+  public Long getChunkSize()
+  {
+    return chunkSize == null ? null : chunkSize.getBytes();
+  }
+
+  public long getMaxResultsSize()
+  {
+    return maxResultsSize.getBytes();
+  }
+
+  public int getMaxTriesOnTransientError()

Review Comment:
   Can we call this `getMaxRetryCount()`? Presumably we'll retry only when reasonable to do so.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest
+{
+  @Rule
+  public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException expectedException = ExpectedException.none();
+
+  private final TestAmazonS3 s3 = new TestAmazonS3(0);
+  private final String path = "resultId";
+
+
+  private S3OutputConfig config;
+  private long maxResultsSize;
+  private long chunkSize;
+
+  @Before
+  public void setup() throws IOException
+  {
+    final File tempDir = temporaryFolder.newFolder();
+    chunkSize = 10L;
+    config = new S3OutputConfig()
+    {
+      @Override
+      public File getTempDir()
+      {
+        return tempDir;
+      }
+
+      @Override
+      public Long getChunkSize()
+      {
+        return chunkSize;
+      }
+
+      @Override
+      public long getMaxResultsSize()
+      {
+        return maxResultsSize;
+      }
+
+      @Override
+      public int getMaxTriesOnTransientError()
+      {
+        return 2;
+      }
+    };
+  }
+
+  @Test
+  public void testTooSmallChunkSize() throws IOException
+  {
+    maxResultsSize = 100000000000L;

Review Comment:
   Help out our poor eyes? Insert underscores as the thousands separator? Or, since this number is really big, cast from a scientific notation representation?



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetriableS3OutputStreamTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonClientException;
+import com.amazonaws.SdkClientException;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.storage.s3.NoopServerSideEncryption;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.easymock.EasyMock;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class RetriableS3OutputStreamTest

Review Comment:
   `Retryable`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942532676


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.

Review Comment:
   The initial idea was to use a gzinputstream for the chunks but dropped that idea in the implementation. 
   Updated the comment.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.

Review Comment:
   The initial idea was to use a gzinputstream for the chunks but dropped that idea in the implementation. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942531269


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream
+{
+  public static final long S3_MULTIPART_UPLOAD_MIN_PART_SIZE = 5L * 1024 * 1024;
+  public static final long S3_MULTIPART_UPLOAD_MAX_PART_SIZE = 5L * 1024 * 1024 * 1024L;
+
+  private static final Logger LOG = new Logger(RetriableS3OutputStream.class);
+  private static final Joiner JOINER = Joiner.on("/").skipNulls();
+  private static final int S3_MULTIPART_UPLOAD_MAX_NUM_PARTS = 10_000;
+
+  private final S3OutputConfig config;
+  private final ServerSideEncryptingAmazonS3 s3;
+  private final String s3Key;
+  private final String uploadId;
+  private final File chunkStorePath;
+  private final long chunkSize;
+
+  private final List<PartETag> pushResults = new ArrayList<>();
+  private final byte[] singularBuffer = new byte[1];
+
+  // metric
+  private final Stopwatch pushStopwatch;
+
+  private Chunk currentChunk;
+  private int nextChunkId = 1; // multipart upload requires partNumber to be in the range between 1 and 10000
+  private int numChunksPushed;
+  /**
+   * Total size of all chunks. This size is updated whenever the chunk is ready for push,
+   * not when {@link #write(byte[], int, int)} is called. This is because
+   * it will be hard to know the increase of chunk size in write() when the chunk is compressed.

Review Comment:
   Updated the description. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] adarshsanjeev commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
adarshsanjeev commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r941141475


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetriableS3OutputStream.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.AmazonServiceException;
+import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
+import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
+import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.PartETag;
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.base.Stopwatch;
+import com.google.common.io.CountingOutputStream;
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream;
+import org.apache.druid.java.util.common.FileUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.IOE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * A retriable output stream for s3. How it works is,
+ * <p>
+ * 1) When new data is written, it first creates a chunk in local disk.
+ * 2) New data is written to the local chunk until it is full.
+ * 3) When the chunk is full, it uploads the chunk to s3 using the multipart upload API.
+ * Since this happens synchronously, {@link #write(byte[], int, int)} can be blocked until the upload is done.
+ * The upload can be retries when it fails with transient errors.
+ * 4) Once the upload succeeds, it creates a new chunk and continue.
+ * 5) When the stream is closed, it uploads the last chunk and finalize the multipart upload.
+ * {@link #close()} can be blocked until upload is done.
+ * <p>
+ * For compression format support, this output stream supports compression formats if they are <i>concatenatable</i>,
+ * such as ZIP or GZIP.
+ * <p>
+ * This class is not thread-safe.
+ * <p>
+ * This class can be moved to the s3 extension as a low-level API,
+ * whereas it currently provides only high-level APIs such as S3DataSegmentPuller.
+ */
+public class RetriableS3OutputStream extends OutputStream

Review Comment:
   nit: I think we can stick to "Retryable" everywhere since that seems to be the current convention



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943792244


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * <p>
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * <p>
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * <ol>
+ * <li>{@code JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);}</li>
+ * <li>// bind the storage config provider {@code binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);} </li>
+ * <li>// Use Named annotations to access the storageConnector instance in your custom extension.{@code @Custom StorageConnector storageConnector} </li>
+ * </ol>
+ * The final state of this inteface would have
+ * <ol>
+ * <li>Future Non blocking API's</li>
+ * <li>Offset based fetch</li>
+ * </ol>
+ */
+
+@UnstableApi
+public interface StorageConnector
+{
+
+  /**
+   * Check if the path exists in the underlying storage layer. Most implementations prepend the input path
+   * with a basePath.
+   *
+   * @param path
+   * @return true if path exists else false.
+   * @throws IOException
+   */
+  @SuppressWarnings("all")

Review Comment:
   I remember IntelliJ inspections complaining. Removed it for now. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943789396


##########
core/src/main/java/org/apache/druid/storage/local/LocalFileStorageConnectorProvider.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.local;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.storage.StorageConnector;
+import org.apache.druid.storage.StorageConnectorProvider;
+
+import java.io.IOException;
+
+@JsonTypeName("local")
+public class LocalFileStorageConnectorProvider implements StorageConnectorProvider
+{
+  @JsonProperty
+  String basePath;
+
+  @JsonCreator
+  public LocalFileStorageConnectorProvider(String basePath)
+  {
+    this.basePath = basePath;
+  }
+
+  @Override
+  public StorageConnector get()
+  {
+    try {
+      return new LocalFileStorageConnector(basePath);
+    }
+    catch (IOException e) {
+      throw new RE(
+          e,
+          "Unable to create storage connector[%s] for basepath[%s]",

Review Comment:
   Adjusted the message a bit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943771820


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;

Review Comment:
   By creds, I assumed that you were talking about the bucket but I later realized that you might be talking about the s3 secret key and access key. 
   All that stuff is injected as part of the AmazonS3 client builder using the the old mechanisms. This PR is not touching how the s3Client is getting built 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r943803621


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ServerSideEncryptingAmazonS3.java:
##########
@@ -43,9 +51,10 @@
 
 /**
  * {@link AmazonS3} wrapper with {@link ServerSideEncryption}. Every {@link AmazonS3#putObject},
- * {@link AmazonS3#copyObject}, {@link AmazonS3#getObject}, and {@link AmazonS3#getObjectMetadata} methods should be
+ * {@link AmazonS3#copyObject}, {@link AmazonS3#getObject}, and {@link AmazonS3#getObjectMetadata},
+ * {@link AmazonS3#initiateMultipartUpload}, @{@link AmazonS3#uploadPart} methods should be
  * wrapped using ServerSideEncryption.
- *
+ * <p>

Review Comment:
   Yup that makes sense :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r942571029


##########
core/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage;
+
+import org.apache.druid.guice.annotations.UnstableApi;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.
+ * For adding a new implementation of this interface in your extension extend {@link StorageConnectorProvider}.
+ * For using the interface in your extension as a consumer, use JsonConfigProvider like:
+ * 1. JsonConfigProvider.bind(binder, "druid,extension.custom.type", StorageConnectorProvider.class, Custom.class);
+ * // bind the storage config provider
+ * 2. binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
+ * // Use Named annotations to access the storageConnector instance in your custom extension.
+ * 3. @Custom StorageConnector storageConnector
+ * <p>
+ * The final state of this inteface would have
+ * * 1. Future Non blocking API's
+ * * 2. Offset based fetch

Review Comment:
   Thanks for catching this. Before this, I was not using the java doc viewer so aggressively but from now on I will try to use that feature more :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] cryptoe commented on a diff in pull request #12874: Introducing Storage connector Interface

Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #12874:
URL: https://github.com/apache/druid/pull/12874#discussion_r952944542


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3OutputConfig.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.HumanReadableBytesRange;
+import org.apache.druid.java.util.common.RetryUtils;
+
+import javax.annotation.Nullable;
+import java.io.File;
+
+public class S3OutputConfig
+{
+  @JsonProperty
+  private String bucket;
+
+  @JsonProperty
+  private String prefix;
+
+  @JsonProperty
+  private File tempDir;

Review Comment:
   https://github.com/apache/druid/issues/12949



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org